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student testing data from a solid evaluation design. Eight schools were 
selected, and their reading data were compared with those of comparable 
public schools. Overall the Edison results were mediocre, small to moderate, 
or not statistically significant. Generally, the results are not as good as 
the results for an average fully implemented Success for All program. Edison 
schools do use the Success for All reading program, operate with a longer 
school day and year, and offer a full-time kindergarten with an academic 
program. However, the Success For All program is only partially implemented, 
half the teachers have less than 5 years of experience, teacher turnover is 
25 to 40 percent, and the class size averages 28 students. The number of 
free-lunch students is declining. By limiting enrollment and having waiting 
lists, the schools' management keeps costs down. In addition to local and 
state funding, investors fund technology, consultants, and staff development. 
Edison has a national reputation and excellent funding, but it may not be 
living up to its claims. (Contains 21 references.) (RKJ) 
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Preface 



The Edison Project is the creation of media entrepreneur Christopher Whittle. Whittle’s 
original plan for the Edison Project was to establish 1,000 for-profit private schools. A series of 
setbacks forced Whittle and his fellow investors to settle on a scaled-back plan that would use 
school district funds to operate a much smaller number of public schools. Whittle now hopes to 
open 200 public schools that will receive the same per-pupil spending as traditional public 
schools, yet still yield a profit. 

Edison spent an estimated $40 million developing its program before its first school 
opened for business, and news accounts have reported that Edison has invested more than $100 
million in its schools. Edison’s pitch to prospective school districts makes several attractive 
claims: Its program will raise student achievement for the same per-pupil expenditure as other 
district schools; provide a longer school day and year; and supply a home computer for the 
family of nearly every student in an Edison school. 

The AFT has followed the Edison Project since its inception, and AFT leaders and staff 
leadership have engaged in an ongoing dialogue with Edison for several years. Christopher 
Whittle has addressed the AFT Executive Council and policy councils. And AFT affiliates in 
two locatioris have negotiated contractual waivers with Edison, smoothing the way for Edison to 
operate schools in those districts. In fact, leaders of the United Teachers of Dade successfully 
lobbied for Edison when a new school board member attempted to terminate the Edison contract 
after one year of operation. 

Not all of AFT affiliates’ dealings with Edison have been harmonious, however. The 
Duluth Federation of Teachers has met with Edison officials and visited Dade-Edison, but was 
not successful in resolving union concerns. Edison opened two charter schools in Duluth in fall 
1997. 



In 1995, Edison opened four elementary schools: in Boston, Massachusetts; Wichita, 
Kansas; Mount Clemens, Michigan; and Sherman, Texas. In 1996, four more elementary 
schools opened: in Dade County, Florida; Colorado Springs, Colorado; Worcester, 
Massachusetts; and Lansing, Michigan. And this past fall, Edison opened elementary schools in 
Chula Vista, California; Detroit, Michigan; Duluth, Miimesota; Flint, Michigan; Southwest 
Independent School District— a small district on the outskirts of San Antonio, Texas— and an 
additional school in Wichita. Edison also operates middle schools in Wichita, Sherman, Mt. 
Clemens, Boston, Worcester, Lansing, and Duluth. The company currently operates 25 schools. 
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Overview of Edison Project Sites 1997-98 



Sites: 


School Name: 

I 


Schools/Date 

Opened 


i # Students 


Size/% School 
District (1993) 


Charter or 
Contract 


Boston, MA 


Boston Renaissance; 
Boston Middle 


1 ES (95); 
1 MS (96) 

1 


660 ES; 
418 MS 


62,407: 1.5% 


j 

charter 


Chula Vista, CA 


Feaster Elementary 


lES (97) 


1,007 


18,240: 18% 


charter 


Col. Spring, CO 


Roosevelt Elem.; 
Emerson MS 


1 ES (96); 
1 MS (97) 


674 ES; 
138 MS 


32,024: 2.6% 


charter 


Dade County, FL 


Reeves Elem. 


1 ES (96) 


1,069 


303,346: .4% 


contract 


Detroit, MI 


Detroit Academy of Arts 
and Science 


1 ES (97); 


705 


172,330: .4% 


1 

charter 


Duluth, MN 


Edison-Central Jr. 
Academy; Edison- 
Kenwood Primary Elem. 


1 ES (97); 
1 MS (97) 


322 ES; 

202 MS 

1 


14282: 4.6% 


charter 


Flint, MI 


Garfield-Edison 
Partnership School; 
Williams-Edison PS. 


2 ES (97) 


476 ES; 
497 ES 


26,615: 2.6% 

! 


contract 


Lansing, MI 


Mid-Michigan Public 
School Academy; Mid- 
Michigan Jr. Academy 


1 ES (96); 
1 MS (97) 


696 ES; 
272 MS 


20,851 


charter 


Mt. Clemens, MI 


MLK Jr. Academy; 
Mt. Clemens Jr. Ac.; 
Mt. Clemens Sr. Ac. 


1 ES (95); 
partial MS (96); 
partial HS (97) 


554 ES; 

347 MS; 

122 HS 

1 


3,356: 25.6% 


contract 


Sherman, TX j 


Washington Elem.; 
Dillingham MS 


1 ES (95); 
partial MS (96) 


460 ES; i 
300 MS 


i 

5,893; 13.4% 

1 


contract 


Southwest ISD, 
San Antonio, TX 


Elm Creek Elem. 


1 ES (97) 


561 


8,462: 6.4% 


contract 


Wichita, KS 


Dodge-Edison ES; 
Jardine-Edison MS; 
Edison Ingalls-Isley PS 


1 ES (95); 

1 MS (96); 

2 ES (97) 


637 ES; 

924 MS; 

684 ES; 293 ES 


47,810: 3.1% 


contract 


Worcester, MA 


Seven Hills Charter 
School 


1 ES (96); 
1 MS (97) 


448 ES; 
216MS 


21,404: 2.5% 

j 


charter 



* Edison Project Student Enrollment Report, 12/5/97. 
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Executive Summary 



The Edison Project offers a mixture of appealing features, some backed by solid evidence 
of educational efficacy and others not. There is also evidence of some demographic and 
financial advantages that are not an acknowledged part of Edison's financial package but are 
nonetheless significant. All told, there are certain features of the Edison Project that would 
predict success in raising student achievement. 

In contrast, there are certain features of the program, as implemented, that might 
undermine Edison’s success. Chief among them are departures from the recommended 
implementation of Success for All, reliance on novice teachers, high teacher turnover, and large 
class size. 

1. Potential advantages of the Edison program 

• In its elementary schools, Edison relies heavily on Success for All (SEA), a successful and 
widely admired reading program designed to raise student achievement in low-performing 
schools with high concentrations of disadvantaged children. SFA was developed by 
researchers at Johns Hopkins University and is now used in approximately 500 schools 
throughput the U.S. It includes daily 90-minute reading periods in small groups of like 
proficiency, extensive one-on-one tutoring in the early grades, family support services, and a 
number of specific staffing and staff development components. 

• Edison expands the time students spend in school with a longer school day and school year 
and a full-time kindergarten that has an academic program. Though these enhancements are 
attractive and popular, their educational effectiveness has not been solidly established. 
However, some studies have found benefits in a good, full-day kindergarten program, and if 
more time in school means more time on task, a longer school day and school year also might 
be expected to improve student achievement. 

• Edison charter schools are, in effect, magnet schools. The parent and student choice that 
such schools entail is often a proxy for highly motivated parents; and that characteristic, in 
turn, can be expected to show up in improved student achievement. The same is true, to a 
lesser extent, of the so-called partnership schools, which Edison manages under contract with 
school districts. In those schools, some out-of-boundary students who wish to enroll are 
permitted to do so, thus creating a “neighborhood magnet.” 

Magnet schools also can exercise some degree of selectivity, which probably explains the 
consistent decline in the number of free-lunch students in many of the Edison schools for 
which we have data.' Since there is strong correlation between student achievement and 
socioeconomic status, fewer poor children at Edison schools would probably increase the 
likelihood of higher test scores. 



' The proportion of students eligible for free or reduced-price lunch in Edison elementary schools has fallen from 76 
to 69 percent in Wichita, from 70 to 64 percent in Sherman, Texas, from 54 to 44 percent in Colorado Springs, from 
64 to 50 percent in Boston, and from 59 to 48 percent in Mt. Clemens, Michigan. 
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Being able to enroll out-of-boundary students also allows Edison to fill every seat by 
limiting enrollment and admitting students fi"om waiting lists. The company can thus make 
the most economical possible use of the school staff and facilities. 

• The first Edison schools have not, in fact, had to make do with the same per-pupil 
expenditures as other public schools. Although Edison schools probably got approximately 
the same local and state fimding as similar local schools, most received additional fimds fi"om 
Edison investors for technology, consultants and staff development, and, sometimes, other 
expenses. Some schools, such as the Boston charter school, have benefitted fi"om substantial 
private donations. In fact, Edison has a person on staff to assist schools in raising 
contributions fi"om outside soiurces. 

2. Potential shortcomings of the Edison program 

• The effectiveness of Success for All has been threatened by the company's failiure to carry out 
the program as developed by SEA researchers. Though reading tutors are central to SFA's 
success, Edison has yet to provide the number of tutors that SFA calls for in high-poverty 
schools, and it spreads reading tutors over all grade levels, fi"om kindergarten through grade 
five, instead of concentrating them in kindergarten through grade two where, SFA 
researchers have found, they are most effective. (That may account for the lackluster 
performance of the Dade County school in 1996-97 in comparison with that of the Edison 
school iri Wichita, where SFA was fully implemented.) Also, except for Dade County, 
beginning in 1997, Edison does not employ the full-time SFA coordinator in each school, 
even though the program insists this is necessary in order to ensiure quality control. 

• Edison relies heavily on relatively inexperienced teachers. Typically, half of the teaching 
force has fewer than five years of experience, in comparison with the national average of 16 
years. 

• Teacher turnover rates in Edison schools are high. The company admits to a 23 percent 
turnover rate, which is twice the national average for turban public schools. At some of the 
Edison schools, turnover is 25 to 40 percent in a single year, as much as triple the public- 
school average nationwide. 

• Class size also tends to be high. With the exception of the small reading groups required by 
Success for All, the average is in the neighborhood of 28 students per class. In some of the 
Edison schools, this is no higher than the school-district average. However, the national 
norm is 24 students, and most experts now agree that elementary school students, particularly 
poor children, benefit fi"om a class size of 15 to 17. 

3. Edison’s Primary Reading Studies. Edison commissioned Robert Mislevy, a 
distinguished researcher, to do reading achievement studies of students in grades K-3. The 
design of these studies was modeled on the standard evaluation used with Success for All 
programs. Mislevy carried out his study in Edison elementary schools in Wichita, Kansas; Mt. 
Clemens, Michigan; Sherman, Texas; and Colorado Springs, Colorado. (He also conducted a 



reading study at Edison’s Boston charter elementary school. However, it had no control group 
and so is not included in our analysis.) 

On the whole, Edison results were mediocre. Kindergarten students were the most 
successful in comparison with control-group students. However, this is not surprising given the 
fact that Edison runs full-day kindergartens with an academic program and the schools attended 
by control-group schools did not. 

Edison first graders did much less well. Students at Wichita performed somewhat better 
than students in the control groups— the effect size was small to moderate. In Mt. Clemens, the 
differences were generally not statistically significant. Furthermore, since no initial achievement 
data were collected, it is not possible to compare Edison students with control group students, so 
test results are inconclusive anyway. At Sherman, the Edison Program showed no effect— that is, 
there was no difference in test results between Edison students and students in the control group. 
At Colorado Springs, results were not good but should probably be considered inconclusive 
because of insufficient data and problems in matching control group students with children 
attending the Edison school. 

Edison shows no signs of continuing the Primary Reading Studies at either Sherman or 
Colorado Springs, so we are unlikely to find out if the students in these schools would have gone 
on to match the achievement levels of youngsters at Wichita and Mt. Clemens. 

4. Success for All results compared with Edison's. Mislevy's evaluations compared 
scores for Edison elementary school students in Wichita, Mt. Clemens, Sherman, and Colorado 
Springs with scores for control group students. However, if you also compare Edison scores 
with scores on a national SFA study carried out by the researchers who originated the program, 
Edison's results are much less impressive. Generally, they are not as good as the results for an 
average, fully-implemented SFA program. It should be noted that the great majority of SFA 
programs are located in public schools that serve poor, minority communities and that student 
populations in these schools are, on average, more disadvantaged than student populations in 
Edison schools. 

5. Other elementary and middle school results 

• The results for Wichita students in grades 3 to 5 are contradictory. Over a two-year period, 
student achievement, as measured by MAT/7, a standardized test of reading and writing 
administered by the school district, has improved dramatically and puts Edison's Wichita 
school well ahead of most comparable schools. In stark contrast, however, the Kansas State 
Assessment shows that Edison students have fallen behind students in comparable schools. 

• After one year of operation, the middle school in Wichita, unlike the more successful Edison 
elementary school in Wichita, demonstrated no gains on the MAT/7 tests or the Kansas state 
assessments. 
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• Over two years, the Texas state assessments show that third- and fourth-grade students in 
Sherman have suffered substantial declines in their math scores and third graders in their 
reading scores. 

• Results in the Sherman middle school are inconclusive because we lack the data to interpret 
them. 

• First-year data for Mt. Clemens students in grades three to five show little overall 
improvement on state assessments, and after two years, standardized test scores improved 
only a small amount. 

• In the Boston charter school, scores for students in grades 3, 4 and 5 have shown 
improvement on MAT/7 tests, though there were irregularities in the way Edison 
admininstered the tests. On the state assessments, third graders scored at the 45th percentile 
in reading, in contrast to the 60th percentile score the same students received on the MAT/7 
the following fall. (It is imheard of for students to register an achievement gain like this over 
the summer.) 

• In Dade County, an early evaluation conducted by the school district concluded that Edison 
students in grades 1 to 6 are doing as well in reading as comparable students in the district, 
but they have fallen behind in math. 

• Early results in Colorado Springs indicate that students in grades 3 to 5 did not improve in 
the school district's standardized test of reading, math, and language as much as either other 
third to fifth graders in the district or third to fifth graders in prior years when the school was 
not run by Edison. 

6. Recommendations. As one might expect at this early stage in the history of the 
Edison Project, the evidence on student achievement is mixed and inconclusive. However, two 
things are clear. There are discrepancies between the record of Edison schools, as measured by 
standard methods of educational evaluation, and the company's sales presentations and 
promotional materials. Edison has exaggerated test score gains and emphasized favorable 
comparisons in order to show Edison schools in the most positive light. In fact, if public schools 
were to use some of Edison’s evaluation methods and modes of presenting data, they would look 
a lot better, too. 

This is unacceptable. Edison should be expected to measure achievement in its schools 
using the same standards that apply to other schools in the districts where the company operates. 
Despite its claims to being a better alternative to regular public schools and a model for public 
education, Edison is obviously confironting the same difficulties in improving student 
achievement as regular public schools. 

Although accountability for the outcomes of private management of a public institution, 
like a school, rests with the private provider, the ultimate responsibility is with public officials 
whose duty it is to protect the public interest. Private management of public schools requires 
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good public oversight, both before entering into an agreement with a private provider and during 
it. To that end, there are a number of things school districts that are considering hiring the Edison 
Project should keep in mind: 

• Imp roving student achievement is at the heart of all the many efforts to improve U.S. 
education, including Edison’s. So school districts should pay at least as much attention to 
Edison’s student achievement data as they do to impressions from site visits to Edison 
schools. 

• School districts should have their own experts-^/wtr/ct officials in charge of program 
evaluation and student assessment— review and verify ^ student achievement data in all of the 
Edison schools. This review should be made public and should include not only Edison- 
administered achievement tests but also state and local assessments. 

• School districts considering hiring Edison should have a timeline that is unhurried enough so 
that everyone involved can give careful attention to the advantages and disadvantages of 
Edison before making a decision. 

• If a school district decides to sign a contract with Edison, officials should either hire 
independent professional evaluators or conduct their own annual evaluation of program 
implementation and student achievement using qualified school-district employees. 

In her article about Edison’s Boston charter school, which appeared in the March 1998 
issue of Phi Delta Kappan, Peggy Farber puts the Edison Project into its perspective as a business 
venture and as one struggling school reform project among many. She directs her caveats to 
members of the media, but they are equally applicable to school districts: 

It is not reasonable to e?q)ect the Edison Project to present the public— its potential 
customers— with a truly comprehensive, objective picture of its schools. Edison officials 
are naturally eager to draw attention to signs of success, which clearly exist. But it is 
essential, especially now, when the idea of a single solution to complex social problems 
has such a strong grip on the American imagination, that reporters give an honest and 
thorough accoimting of what it's like inside schools that operate beyond the reach of almost 
all local and state agencies. The evidence from Boston Renaissance suggests that the 
Edison Project is struggling— succeeding in some areas, stumbling in others— to improve 
schools for students. And the same can be said for countless reform efforts across the 
coimtry. 

To a large extent, Edison’s popularity depends on the perception that public schools are 
dysfunctional and helpless to change and that anyone with a fix to offer must be able to do better. 
Before buying into this idea, public officials and the people who report and comment on education 
should look at some of the reform efforts that are demonstrably— that is, using accepted measures 
-changing public schools for the better. This report will comment on one of them- Success for 
All— and to point out that, thus far, Edison students who are following the SFA program are not 
doing as well as SFA students in non-Edison schools where SFA is fully implemented. This is 
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true despite the fact that Edison students are generally more advantaged and have the benefit of 
the rest of the Edison program. 

The point is that Edison is one reform effort among many. Its more polished public 
relations have ensured it a national reputation, and since it is a for-profit business, it also has 
more money behind it. Will the Edison Project be able to live up to its promises? We’ll have to 
wait and see. In the meantime, there are unheralded programs that are already living up to their 
claims, and people who are looking for ideas— or good news about education— should be giving 
them the same kind of attention that Edison gets. 
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I. Why This study? 



When Edison Project representatives pitch the company's program to school districts, 
they offer a very attractive-sounding package: a longer school day and school year; a home 
computer for the family of nearly every student; a full-day kindergarten with an academic 
program; and a strong curriculum— all of which, they say, will lead to documented and significant 
improvements in student achievement. The price appears to be right, too. Far fi-om requiring a 
big financial commitment, Edison promises that it will provide its program for the same per- 
pupil spending as regular public schools— with enough left over to yield a profit for investors. 

When school districts first contracted with Edison, they had to do so largely on the basis 
of the company's plans and promises. Now, there are two years of data fi"om the four schools 
Edison opened in fall 1995 and early results fi"om schools that opened in fall 1996. What do the 
data indicate? 

Although Edison rightly cautions against expecting too much too soon, the company also 
makes ambitious claims about the success of its program, based, it says, on these early results. 

Its 1997 annual report on school performance speaks of "a broad trend of significant 
improvement" and says that the Edison program is improving achievement for 70 percent of 
students and failing with only 5 percent, with differences for the other 25 “not large enough to 
draw firm conclusions.” (http://www.edisonproject.com/amiualfTame.html) Beimo Schmidt, the 
president of the Edison Project, calls the results: "uniformly positive... where we can compare the 
achievement of Edison students to similar students in other public schools or measure the current 
performance of Edison students against their past achievement." 

But though Schmidt invokes the kinds of comparisons needed for an objective assessment 
of how well Edison is doing, few independent studies of results in Edison schools exist. Except 
for the limited evaluations of the Success for All program in fom schools conducted by Robert 
Mislevy of the Educational Testing Service and a school district evaluation in Dade County, no 
independent evaluations of Edison schools are being carried out or, apparently, are planned. And 
there are no independent studies analyzing and synthesizing the available data on the 12 Edison 
schools that had been in operation for at least a full school year at the time this report was 
compiled. So those interested in following the progress of the Edison enterprise have had to fall 
back on the company's promotional materials and its rather skimpy and selective analyses of 
student achievement data. 

This report aims to offer a solid and comprehensive basis for evaluating Edison's progress 
thus far by gathering together and analyzing the available student achievement data fi'om all the 
Edison schools. And since, unlike other studies, it integrates data for all of the achievement 
measmes used in assessing Edison schools, it can provide a fi'amework for futiu'e evaluations. 
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II. How We Did This Study 



This study focuses almost exclusively on student achievement in selected Edison schools 
through an analysis of test score data. It does not deal with student or parent or teacher 
satisfaction, nor does it contain information gathered from school visits. These perspectives are 
important, but we prefer to focus on the bottom line of student achievement because that is the 
bottom line and because interest in the Edison Project turns primarily on its capacity to improve 
student achievement, particularly in relation to regular public schools. Moreover, penetrating test 
score data or claims is often a difficult business for the public and policymakers and even for 
some school administrators; the more sources of analysis they have, in addition to the reports 
provided by Edison, the better their understanding and decisions can be. 

This focus on student achievement determined our criteria for which of the 25 existing 
Edison schools (as of the 1997-98 school year) to examine. The first criterion was track record, 
that is, how long the school was in operation. The second, related criterion was solidity of 
student testing data and evaluation design. Those criteria excluded the 13 Edison schools opened 
in 1997. Although there are certainly impressions to be gleaned from a first or single year of a 
school’s operation, it would be premature for us (or Edison) to draw conclusions about student 
achievement after only one year. 

Conversely, those criteria led us to focus primarily on the four elementary schools Edison 
opened in 1995-96 in its first year of operation: the Dodge-Edison school in Wichita, Kansas; 
Martin Luther King, Jr. Academy in Mt. Clemens, Michigan; Washington Elementary School in 
Sherman, Texas; and Boston Renaissance in Boston, Massachusetts. We also examined the eight 
schools Edison opened in 1996-97, its second year of operation. These newer schools were 
automatically less solid candidates on the criterion of track record, but Edison includes them in 
their most recent report, so we examined their student test data and evaluation designs (our 
second criterion) to see if they allowed a solid or promising early assessment of student progress; 
if they did, we included the schools in our student achievement analysis. 

This examination of the eight schools opened in 1996-97 yielded four schools that met 
the criterion: Roosevelt Elementary in Colorado Springs, Colorado; Dillingham Middle School 
in Sherman, Texas; Jardine-Edison Middle School in Wichita, Kansas; and Reeves Elementary in 
Dade County, Florida. We did not, however, conduct an analysis of student achievement in 
Reeves/Dade because the school district is already doing so using an evaluation design that we 
believe is consistent with the evaluation principles enunciated later in this chapter. Instead, we 
report some results from the school district’s preliminary evaluation. Analyses of student 
achievement in the other three schools are included in our report. 

The schools that receive little of our attention are Seven Hills Charter School in 
Worcester, Massachusetts; Mid-Michigan Public School Academy in Lansing, Michigan; Mt. 
Clemens Junior Academy in Mt. Clemens, Michigan; and Boston Middle School in Boston, 
Massachusetts. The chief reason is the inadequacy of their baseline for measuring student 
achievement. Baselines for these schools were established using spring testing data, almost a 
year after the schools opened, which means that the Edison program had had nearly a full year’s 
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impact on student achievement. Thus, although there are student testing data, without a proper 
baseline, it is difficult to tell what these data mean (and what future data will mean). We offer a 
fuller discussion of the baseline data issue later in this section. 

Three of the four schools in the above category are also charter schools. And, in general, 
charter schools get less attention in our report because they fall short on our second criterion, 
solidity of evaluation design and student testing data. In the case of charter schools, the typical 
evaluation problem is that there are no comparison schools or students. Combined with the 
potential for student selection bias and/or the self-selection of parents, determining a program 
effect with any confidence is very difficult 

In sum, then, our report offers student achievement analyses for eight Edison schools 
(including the school district evaluation in Dade County) and focuses most heavily on the four 
schools in this group with the longest track record of operation - Edison’s debut schools. Other 
schools may be mentioned in a context relevant to student achievement, but if we did not have 
data or data that met our criteria, we could derive little about student achievement. 

1. Primary Reading Studies. The four Edison schools we focus on most heavily also 
received special scrutiny from Edison. Edison contracted with Robert Mislevy, a distinguished 
statistician, to conduct “Primary Reading Studies” modeled after the highly regarded evaluation 
design used to ascertain student achievement in Success for All elementary school programs. 
(Note: Although Mislevy works for the Educational Testing Service, his work for Edison was 
not done under its auspices.) Success for All also happens to be an important component of 
Edison’s own elementary school program, so the evaluation model for the Primary Reading 
Studies offered a particularly good fit. 

Our report discusses the results of Mislevy’ s Primary Reading Studies and extends them 
further by offering comparisons between the Edison schools and Success for All results in 
regular public schools. However, our analysis of the Primary Reading Studies excludes Boston 
Renaissance, one of the original four Edison schools, because the study had no control group. 
Without a control group, no program effect can be determined, so the Boston study is rarely 
discussed in this report. We do, however, discuss the only Primary Reading Study conducted in 
the second batch of schools opened by Edison (1996), and that is in Roosevelt Elementary in 
Colorado Springs. All told, then, our analysis of the Primary Reading Studies is limited to 
Dodge-Edison/Wichita; Martin Luther King, Jr. Academy/Mt. Clemens; Washington 
Elementary/Sherman; and Roosevelt Elementary/Colorado Springs. 

2. Other data sources and data displays. Our study follows the principle of multi- 
indicator analysis of student achievement, that is, using as many sound sources of student 
achievement data as is possible. We therefore report both test scores and gains in test scores and, 
whenever possible, compare Edison school test results to home district test score averages, state 
averages and results in comparable schools. 

Edison might argue that the only relevant measure is gain in test scores. We agree that 
gains are a relevant measure, but so are the other components of multi-indicator assessment. 
Moreover, since Edison generally declares that it will be held accountable for performance on all 
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state and local assessments—and because regular schools are examined and held to account on 
this basis--a multi-indicator analysis offers school districts the kind of information they are 
accustomed to and need. 

To further encourage readers to examine student achievement from a variety of 
perspectives, we generally present all testing data for all grades in all years, which Edison 
usually does not. Readers may then either follow a single cohort of students over time from one 
grade to the next, which is the predominant way Edison displays data, or they may compare 
progress at a single grade level from one cohort to another, which is the way student achievement 
comparisons are usually conducted; district, state and federal Title I assessment reports almost 
always compare progress at a single grade level over time, and this is how national and state 
assessment norms are established. 

3. Baselines. The choice of baseline for measuring progress in student achievement is a 
critical one. Typically, when trying to determine the effect of a program, the choice of baselines 
focuses on identifying the point in time when the program has least affected the student. The 
ideal baseline would be an individual student test score from the spring immediately preceding 
the introduction of a new program, which in schools is typically the fall. A very poor— indeed, in 
our judgment, unacceptable— baseline would be scores from tests taken in the spring after the 
program was introduced because that program would have had nearly a full school year’s impact 
on lowering or raising student achievement. Subsequent evaluations would then be comparing 
the effects of program X to program X, rather than the effects of program X to pre-program X. 

Unfortunately, good baselines are not always available for Edison schools. As discussed 
earlier, our analysis did not fully treat four second-year Edison schools because their baseline 
student achievement data were constructed from tests taken in the spring after conversion to an 
Edison school. Again, the ideal baseline would be student test scores from the spring 
immediately preceding enrollment in an Edison school. 

An alternative is a baseline of test scores from the early fall of the first year of 
conversion to (or enrollment in) an Edison school, but that baseline has serious shortcomings. 
New schools are often chaotic, and the students may not be prepared to take an early fall test. 
Moreover, teachers and administrators are aware that the test is only for the purpose of 
establishing a baseline for progress and that no one will be held accountable for the results; the 
test doesn’t “count.” And while other schools in the district are being held accountable for the 
prior year’s school performance, the initial test at an Edison school (or any new public school) 
carries no such burden. 

Yet another, and preferable, alternative is to constmct a baseline consisting of pre-Edison 
average test scores in the same school, provided that the student body composition does not 
change substantially in the shift to Edison management or provided that such changes are 
controlled for statistically. We were able to do that for four Edison schools: DodgeAYichita; 
Washington Elementary/Sherman; Martin Luther King, Jr. Academy/Mt. Clemens; and 
Roosevelt Elementary/Colorado Springs. In every case where we examined pre-Edison test 
scores, we conducted an examination of student demographics and/or the percentage of students 
coming from the neighborhood attendance zones. Although the demographic profile of the 
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students prior to the school’s conversion to Edison was a little different from the student profile 
under Edison management, simple comparisons always favor Edison; in all cases, the students in 
the school prior to Edison were slightly more disadvantaged. Thus, if anything, the baseline 
constructed from pre-Edison school average test scores would be biased toward a positive impact 
on student achievement from the Edison program. 

4. Data sources. Student achievement data for this report came from published and 
unpublished reports, information distributed by Edison at client meetings, and from Internet 
docmnents of school districts and states. Some information also came from end-of-year school 
reports prepared by Edison. We also gratefully acknowledge Edison’s sharing of student 
achievement data with us. This information was particularly useful for examining the Edison 
charter schools where Edison conducts its own testing. In most instances, Edison provided data 
from the test publishers, indicating the nrnnber of students tested, test dates, test forms, and other 
relevant information. 

We are encouraged by this cooperation on Edison’s part. Although student achievement 
data from public schools are public information, some other companies in Edison’s position are 
slower to recogmze this. By providing us with information that is lacking from the company’s 
usual presentation of student achievement data, Edison is signaling an acknowledgement that, 
when you educate students at public expense, there is an obligation to make public information 
that is typically a matter of public record and interest. We applaud Edison for this and urge the 
company to continue down that road. And we urge the public, most especially local and state 
public school officials, to ask for and assess all the data pertinent to Edison and other publicly 
funded schools. Indeed, stimulating such inquiry, so crucial for the cause of education reform 
that works, is the point of this study. 
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III. The Edison “Advantage” 



What kinds of advantages does Edison point to when it offers its program to school 
districts and, indirectly, to parents? Success for All, the well-respected elementary school 
reading program that is especially tailored for disadvantaged children, is certainly one of them. 
Some of Edison's other program advantages may have more popular appeal than substance. The 
longer school day and school year, for example, are very attractive to the public and many 
parents, and this is certainly an advantage from a marketing point of view. However, the 
evidence that they improve student achievement is not conclusive. On the other hand, there are 
some unacknowledged financial advantages— for example, being able to dip into corporate 
coffers, at least for now— and some demographic advantages, which are unrelated to Edison's 
academic program but give Edison schools some advantages over the regular public schools with 
which they are competing. 

1. Success for All. The centerpiece of Edison elementary schools is the highly regarded 
Success for All program. As of the 1996-97 school year, SFA is being implemented in 
approximately 500 schools in more than 126 districts in more than 37 states in all parts of the 
United States.^ When fully implemented, SFA includes the following components: 

• Preschool and kindergarten. Whenever possible, a half-day preschool program is 
provided for all four-year-olds. The program emphasizes language development, reading 
readiness, and positive self-concept. A fiill-day kindergarten program (some programs 
have only half-day) continues the emphasis on language and math concept development. 

• 90-minute reading periods. During 90-minute reading periods, students work in small, 
homogeneous groups of 15 students. The groups cut across grades and contain students 
who are all reading at the same achievement level. 

• Frequent assessments. Every eight weeks, reading teachers assess student progress using 
personal observations and curriculum-based tests. Teachers use assessments results to 
identify students who are falling behind and need extra help and tutoring, as well as those 
who are progressing quickly and should be placed in a higher performance group. 

• Reading tutors. In grades 1-3, specially trained, certified teachers who are reading 
specialists work one-on-one with any students who are failing to keep up with their 
classmates in reading. Specially trained paraprofessionals may also be reading tutors. 
Priority for tutoring is given to first-grade students, with each tutor providing help for 9 
to 1 1 students. The one-on-one tutoring occurs in 20-minute sessions outside of the 
small-group reading sessions. Tutors also act as regular reading teachers during the 90- 
minute reading periods. 



^ For more information on SFA see: http://successforall.com/componeiits.html and 
http://successforall.com/considerations.html. 
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• SFA facilitator. The full-time facilitator, a member of the school staff who is released 
from regular classroom or administrative tasks, is responsible for making sure SFA is 
correctly implemented. The facilitator oversees details of the program, such as 
scheduling, professional development, classroom implementation, and assisting teachers 
and tutors with problems, including student behavior problems. He is responsible for 
organizing the regular student assessments and changing reading group assignments as 
needed. 

• Family support team. The family support team promotes parental involvement in the 
school by providing information, organizing school-related activities, and conducting 
workshops for parents. An important function of the team is to intervene to help solve 
behavior and other problems. The team is composed of the principal or assistant 
principal, SFA facilitator, social worker, and other personnel. 

Extensive research shows that SFA, when fully implemented, gets very good results, 
particularly with low-income children (Slavin et al.. 1996a, b; Slavin and Madden, 1993). 
Statistically significant positive effects have been found from grades one to five, with 
especially large gains for students most at risk for failure. These effects have also been 
shown to be cumulative. While first-grade SFA students are about three months ahead of 
control students in reading, by fifth grade, they outscore control students by an average of a 
full grade level. Bilingual students and students in the lowest quartile of their grades average 
even higher gains, with effect sizes of 1.00 or more. (According to SFA standards, .50 is a 
moderate effect size, and .80 is considered large. See the discussion and table of effect sizes 
on page 27. See also http .'//success fora ll.com/sumresc h . h tm 1 and http://successforall.com/Articles.htm) . 

2. Longer school day. The evidence in favor of the longer school day (like the longer 
school year and the full-day academic kindergarten) is far less conclusive than that favoring SFA, 
in part because they are ideas rather than carefully developed and tested programs. There is no 
doubt, however, that parents find them attractive, so they constitute a marketing advantage, at the 
very least. 

The Edison school day is about one hour longer than a typical public school day. While 
common sense would seem to argue that longer school days promote greater student 
achievement, common sense also tells us that this would depend on how the time is spent. Other 
nations, including Germany and Japan, have shorter school days than the U.S. at all grade levels, 
and especially in elementary schools. 

3. Longer school year. The Edison program adds about one month to the school year. 
Although school years are longer in some other countries with successful education systems, 
school days are shorter in these countries. U.S. students, all told, already spend more time in 
classrooms than students in almost every other country including Germany and Japan. As with 
the longer school day, adding time is not sufficient, in and of itself, to raise student achievement; 
adding time on task probably is. 
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4. Magnet school effect. Like the financial advantages enjoyed by Edison schools, 
which this report will also take up, the magnet school effect is a characteristic of the Edison 
operation, rather than something the company advertises. However, both of these characteristics 
constitute considerable advantages for the Edison schools. 

In Edison schools, at least some of the students come fi'om outside the neighborhood 
attendance zone, and no neighborhood student must attend. Thus, Edison is able either to recruit 
students or limit enrollment in order to establish their optimum class size (about 28 students) and 
a relatively large school (usually around 600 students in grades K-5). The Edison schools in 
Wichita, Mt. Clemens, Colorado Springs, and Sherman are essentially “neighborhood magnet 
schools.”^ Indeed, in Wichita, Edison's most successful schopl to date, nearly half the students 
come from outside the neighborhood. The Edison charter schools in Boston, Worcester, and 
Lansing have no neighborhood attendance boundaries. 

This arrangement is desirable from a financial point of view— full classes and full schools 
allow Edison to make the most efficient use of the staff and the physical plant.'* But there is 
another advantage, as well: Although out-of-boundary students and those who attend charter 
schools are selected by lottery or on a first-come-first-served basis, the powerful effects of self- 
selection remain. Parents who go to the trouble of enrolling their children in schools outside 
their neighborhood are more likely to value achievement in school and to provide their children 
with the encouragement and help to succeed. And their children are more likely to be successful, 
regardless oTthe program a school offers. Having a significant number of students like these is 
an advantage for Edison, as it would be for any school, not the least because the effects of family 
and environment are likely to be attributed to the effects of the Edison program. 

5. Financial advantages. One of Edison's trademark claims is that its program can 
educate students for the district’s average per-pupil expenditure while offering many advantages 
that public schools cannot match. This notion of doing more with the same amount of money is 
(understandably) attractive to school districts and taxpayers. But, when you look at the data, 
Edison is having it both ways. While claiming to offer a school district extra value for the 
current per-pupil expenditure, the company is, in fact, spending more money per pupil than 
comparable schools in districts where they have set up shop. 



^ Some of Edison’s magnets or neighborhood magnets draw from a much larger pool of families than others. In 
Dade County, the Edison school is one choice in a three-school attendance zone. Mt. Clemens has only two 
elementary schools, one of which is an Edison school. Mt. Clemens’ only middle school — of which Edison runs a 
part — recraits significant numbers of students from other school districts, thanks to Michigan’s open enrollment 
law. 

^ One reason neighborhood schools do not have the waiting lists that Edison boasts of is that they have to enroll 
every child who resides in the neighborhood. Edison schools are not under this obligation. Even the Edison school 
in Colorado, which serves mostly students from the neighborhood, turned away dozens of neighborhood children 
because no space was available after Edison admitted several dozen out-of-neighborhood children (“Turnover 
Shrinks Edison Waiting List,” Colorado Springs Gazette Telegraph, September 10, 1996). The students who were 
turned away were bused to out-of-neighborhood schools, which had to increase class size for a few weeks and put 
up with related educational disraptions. 
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The favorable charter-school funding laws in Michigan and Massachusetts give the 
Boston, Worcester, and Lansing schools more money than comparable schools in their districts. 
In Boston, for example, the Edison charter school gets per-pupil funding that includes a generous 
allowance for all types of special education students, including the orthopedically handicapped 
and the mentally retarded. But since Edison has hardly more than half as many special education 
students as the other local schools, and most of them receive most of their education in regular 
classrooms, the excess special education money is available for use in the regular program.^ 

In some school districts, the contracts with Edison have been carefully written to match 
the funding with the type of student enrolled in the school, so there are no extra public dollars. 
However, in cases like these, Edison’s investors have compensated with substantial investments 
in technology and other corporate support, including money for out-of-town staff development 
and travel, on-site training by corporate officials, and the funding of start-up costs. In Sherman, 
Texas, Edison spent 12 percent more than its $1.8 million budget, and that does not count the 
$1.5 million in start-up costs (Dallas [Texas] Observer, August 15-21, 1996). 

In Wichita, the expenditures budgeted by Edison did not include the company’s corporate 
contribution to technology and facility upgrades, which total $1.5 million, or about $300,000 per 
year over the five years (including $370,000 for home computers). Little or none of the money 
for these upgrades is being paid directly by revenues from the school district. 

But even without the corporate funding, Edison spends more than the per-pupil costs on 
which it claims to run its Wichita school. The company advertises that it ran its Wichita school 
on $3,700 per pupil in 1995-96, but the money the school received from the state and federal 
government for special education and Title I raise the per-pupil spending to about $4,000. 
Moreover, food service and transportation (still provided by the district) are not included in 
Edison’s calculations. Edison pays for three additional buses for some of the out-of- 
neighborhood students, but the school district also deducts a 3 percent overhead charge in the 
process of calculating the total contract amoimt. Transportation, food service and district 
overhead would add about 10 percent to the $4,000 per-pupil cost, raising the total to about 
$4,400. 



In Colorado Springs, Edison obtained the highest funding allowed under the state charter 
law, 104 percent of the district's’average per-pupil cost, or $4,472 per pupil. A school-district 
cost analysis, which was part of the charter application, estimated that the district would lose 
$2.9 million in money that went to the Edison charter but would “avoid” only $2.2 million in 
costs. The $700,000 difference is the extra cost of the contract and represents resources that 



’ Seven Hills, Edison’s charter school in Worcester, Mass., provides another example of the same kind of thing. 

This charter gets the same funding as Worcester public schools. However, in 1996-97, only 14 percent of the 
children attending were considered special-needs students, compared with 20 percent in the public schools. 
Moreover, 70 percent of these students were in the least serious special education categories (prototype 502.1 and 
502.2), and the school had no very handicapped children. During the first half of the 1996-97 school year, 21 
special education children from the school returned to public schools, two-thirds of them in moderately handicapped 
categories (prototype 503.3 to 503.41). Parents were told that the school could not adequately accommodate their 
children’s learning disabilities {Worcester Telegram and Gazette, August 30, 1991). 
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would otherwise have gone to the other schools and students in the district. 

Since the Edison advantage of being able to fund its programs with more than the average 
per-pupil expenditure available to most schools in its districts is not widely known or 
acknowledged, the company is gamering an enviable and beneficial reputation for efficiency. 

But will the consequences of this discrepancy between reputation and reality get Edison into 
financial trouble? Edison estimates that it can become profitable if it runs more than 100 
schools. On the other hand, Edison faces the unforeseen complications that attend running 
schools in many different jurisdictions with many different mles and regulations and funding 
schemes, and this may eat into the company's profits. And if the company is caught in a crunch 
between its legitimate interest in turning a profit and the public’s equally legitimate interest in 
having students receive the full program it paid Edison to deliver, which interest will prevail? 
There is no evidence with which to answer this question. But both the best and worst cases of 
private management or provision of public services provide ample reason to raise the question. 
What we can say at this point is that, much as public school districts sometimes cut comers, so, 
too, has Edison. 
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IV. Cutting Corners 



The Edison program is expensive. A longer school day and school year, computers for 
most of the students, training for school staff in new educational programs-- all of these things 
cost money. At this point, Edison is still using corporate funds to help underwrite some of its 
program's costly trademarks, but the company's announced goal is to make a profit for investors 
while spending the same average per-pupil dollars as regular public schools. 

How will it manage? The company says, by taking advantage of the efficiencies possible 
in the nationwide chain of schools it hopes to build. In the meantime, there are indications that 
Edison is cutting comers by hiring beginning teachers, who are at the bottom of the salary scale; 
upping class size; limiting special education, wherever possible, to children who can learn in a 
regular classroom; and skimping on the implementation of Success for All, the reading program 
that is the centerpiece of its elementary education program. 

Of course, it would be unfair to draw conclusions based on limited evidence. We 
acknowledge the evidence is limited, and we use the word “indications” advisedly. But since 
Edison is asking school districts to make important financial and educational decisions that are 
also based on limited evidence, it is fair to point out signs of practices that could jeopardize the 
quality of Edison's program and therefore its results with students. 

1. Inexperienced and low-paid teachers. It should be no surprise to hear that a great 
deal of research supports what common sense also tells us: Good teaching makes a big 
difference in student achievement. As is the case with other professions— medicine, law, 
architecture, etc.— experience is an important component of a teacher's professional expertise. 
According to the National Commission on Teaching and America’s Future (Darling-Hammond, 
1997), “Studies discover again and again that teacher expertise is one of the most important 
factors in determining student achievement, followed by the smaller but generally positive 
influences of smaller schools and small class size.” Of course, experience and expertise are not 
identical, but the first is often a reasonable proxy for the second. In an analysis of 900 school 
districts in Texas, Ronald Ferguson (1991) found that teacher expertise— as measured by scores 
on a licensing examination, master's degree, and experience— accounted for about 40 percent of 
the variation in students’ reading and math achievement, more than any other single factor. 
Ferguson and Ladd repeated the analysis in Alabama and found significant influences of teacher 
qualifications (1996). But Edison's practice of hiring a large number of beginning teachers 
suggests that getting a skilled and experienced staff takes second place to keeping costs down. 

It's not that Edison hires teachers on the cheap. In Edison schools, teachers' salaries are 
pretty close to what they would be elsewhere in the district for the same level of experience.^ 



* In Wichita, Dade County, and Colorado Springs, beginning salaries are linked to the union-negotiated salary 
schedule, but individual teachers usually negotiate their own salaries as well as any subsequent pay increases. In 
other charter schools, begiiuimg salaries come close to starting salaries in surrounding public schools. However, 
Edison has the power to terminate the employment of any teacher for any reason, including replacement by a less- 
expensive teacher for economic reasons. 
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And Edison's differentiated staffing means that some people--lead teachers with less than five 
years of experience, for example--do much better financially than they would in the other district 
schools. 

The real point is that when Edison hires the staff for its schools, it tends to favor 
beginning teachers over experienced ones. Although most educators would concur that it is not 
good for students to have a majority of inexperienced teachers, this policy does keep a 
company’s personnel costs down: 

• In 1995-96, only 12 percent of the teachers in Edison’s elementary school in Wichita 
had a master’s degree compared to 47 percent in the rest of the district, and they averaged 
5.6 years of experience compared to the district average of 10.2 years (WSU, 1996). 

• Some of the lead teachers in Edison’s elementary school in Wichita had as little as 3 
years of experience and would still be considered new teachers in any other school. 

• At Edison’s Wichita middle school in 1996-97, 55 percent of teachers had 5 or fewer 
years of experience (“Quarterly Progress Report, Jardine Middle School,” October 25, 
1996). 

• In Dade County in 1996-97, only 10 percent of teachers were new to the profession, but 
the average teacher had only four years of teaching experience, and only one in four had a 
master’s degree. 

• In Edison’s Sherman elementary school in 1995-96, 28 percent of teachers were new 
teachers compared to 7 percent in the remainder of the district, and 65 percent had five or 
fewer years of experience compared to 30 percent elsewhere in the district (TEA, 1997). 
The following year, 69 percent of Sherman teachers had fewer than five years of 
experience. 

The financial impact of this hiring policy is clear from Wichita, Sherman, and Mt. 
Clemens. Even factoring in the 10 percent extra pay Wichita-Edison teachers received for their 
longer school day and school year, personnel costs were 20 percent lower than the average costs 
for school district personnel (Wichita State Evaluations, 1996). In Sherman, the 1995-96 average 
salary was no higher for Edison teachers than for other district teachers, despite Edison’s longer 
school day and school year (TEA, 1997). And in the Edison school in Mt. Clemens, the average 
teacher salary dropped from $55,000 to $45,000 when Edison began managing its school 
(http://www.mde.state.Tni.us/cgi-bin/Rf/reports/MSR97/M/50160;'B0049.TXT) . 



2. High teacher turnover. Most educators also would concur that a high teacher 
turnover rate in a school is not good for students. In fact, this is a problem that surfaces often in 
discussions of urban schools, and it is a problem that has emerged in Edison schools to an even 
greater degree. 

Edison selects all the teachers for its schools, and when a school converts to Edison 
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management, only a few of the teachers from the pre-Edison school are likely to be left; 
presumably, Edison teachers are there because they want to be and because Edison thinks they 
can do the job. Edison admits to a turnover rate of 23 percent across all its schools, but it is 
higher in some. In Colorado Springs, there was a 27 percent turnover at the end of the first year 
(Springs Magazine, August, 1997). The situation in Dade Coimty was even more severe; in its 
Jime 1997 year-end report, Edison estimated that at least 20 of 54 teachers—nearly 40 percent— 
were not coming back. And a report submitted to the Massachusetts Department of Education 
indicated that 3 1 of the teachers at the Boston charter school would not be returning— a 44 
percent turnover. 

This is all the more striking because, if teachers survive the first five years, teaching is 
not, generally, a high turnover profession. U.S. Department of Education figures indicate that 
approximately 13 percent of U.S. teachers either leave their current school for another or leave 
teaching altogether each year. 

High turnover can cause (and/or reflect) poor staff morale, even in a school with an 
established program. When a new program is being instituted, high turnover is likely to be 
especially destructive. Teachers who are still relatively inexperienced in a demanding new way 
of doing things— like, for instance, the Success for All program— are also faced with new 
colleagues who are totally imfamiliar with what is required of them. In circumstances like these, 
the program-and the students- are likely to suffer. Edison’s response to the resignation of nearly 
40 percent of the teachers in Dade Coimty indicates that the company itself recognizes the impact 
of high turnover: Edison pledged to invest in training, as though Dade were a first-year school. 

However, the company’s practice of hiring mostly new teachers may lead inevitably to 
high turnover, with all the resulting problems. National figures show that 50 percent of new 
teachers leave the profession, at least temporarily, within the first five years. By choosing this 
method of economizing, Edison is putting its program at risk. 

3. Large class size. Small classes are very expensive, as critics of current efforts to 
reduce class size always point out. Raising class size, which few policy makers ever suggest, is a 
big money saver. And it is the second policy that Edison is apparently adopting in a number of 
its schools— at least unofficially. Except in reading, where Edison, following the Success for All 
model, uses small reading groups and tutors for children who are having particular problems, 
average class size is high by national standards and by the standards of most localities where 
Edison operates. 

The average elementary school classroom in the United States contains about 24 students 
and has changed little in size in 30 years. Middle-school class size is about 25 students. Of 
course, there is great variation among classrooms within a school district and among school 
districts. Over the same period of time, pupil-teacher ratios have steadily declined to the current 
figure of approximately 17 pupils per teacher. This decline is largely the result of increased 
services to special needs students, which have enlarged school staff without reducing class size 
in regular classrooms. 
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At first glance, Edison appears to provide a rich complement of staff, with support 
teachers for language, art, music, and physical education. However, except for reading, where, 
following the Success for All model, Edison reduces reading groups to 15 students, classes at a 
typical Edison school contain around 28 students. In Colorado Springs, there were up to 30 
students in a class (Springs Magazine, August 1997). And despite the initial publicity about 
classes of 21 students at Edison’s Boston charter school, the class size grew to 28 (The Nation, 
September 28, 1997). Some schools have a slightly lower average class size— in the range of 26 
to 28 students (in Dade Coimty, Mt. Clemens, the Lansing (Michigan) elementary school, and 
the Boston charter school). In Sherman, class-size targets are about 24. 

• The 1995-96 year-end report for the Wichita elementary school says that reading groups are 
small but that other classes, except for kindergarten, average 27 to 28 students. The pupil- 
teacher ratio of 20 was higher than any of the eight comparison schools identified by the 
Wichita State University evaluation (1996). (The comparison schools ranged from 17.1 to 
19.3.) School district data for comparison schools identified in this report also show that 
Edison had fewer teachers per 100 students (5.0) than any of the comparison schools, which 
range from 5.3 to 7.1 (see table Al). 

• In Dade Coimty, in 1996-97, the year-end report said the reading classes in Edison’s 
elementary school averaged 22 students, with other classes averaging 27 to 28 students. 
Approximately 1,100 students were served by 41 regular teachers and 16 special teachers (3 
art, 3 music, 3 physical education, 6 language, 1 special education) for a pupil-teacher ratio of 
19 to 1. 

• In the year-end report on the Mt. Clemens middle school, Edison describes “an iimovative 
scheduling [that] enables students to have reading and math in classes of 25 and 22 students 
respectively. Other classes average 28 students each.” Classes this size are hardly 
remarkable, given that the U.S. average for middle schools is about 25. However, the pupil- 
teacher ratio does match the U.S. average of 1 7 to 1 : 3 12 students were served by 10 regular 
teachers and 8 special teachers (1 art, 1 music, 2 physical education, 2 language, 2 special 
education). 

• Edison’s 1997 year-end report at the middle school in Wichita says that the average class size 
is 26, with ESL (English as a Second Language) classes being somewhat smaller. 
Approximately 30 regular teachers and 14 special teachers (1 art, 4 physical education, 3 
language, and 4 special education) serve approximately 936 students for a pupil-teacher ratio 
of 21.3. With 4.7 teachers per 100 students, the school is well below the other four middle 
schools in the district that have a similar percentage of students qualifying for free or 
reduced-price lunch. In these schools the number of teachers per 100 students ranges from 
5.9 to 7.0. 

4. Limiting special education to children who can spend most of their time in 
regular classes. Many parents of special education students, who are eager for their children to 
learn in the same classes as other youngsters their age, would applaud this Edison policy. 
However, the range of children classified as special education students is enormous, from those 
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with mild learning disorders all the way to children who are profoundly disabled or suffer from 
severe emotional disturbances. And there is considerable dispute about whether all special 
education children, regardless of their disability, should be educated in regular classes. What no 
one disputes is that mainstreaming special education students— and particularly children with 
severe problems— is less expensive than providing them with small classes and highly trained 
special education teachers. 

Is Edison's special education policy, in part at least, a way of saving the company money? 
That is what Edison president, Benno Schmidt, said at a Smith-Barney conference for education 
investors that was held in New York City on May 20, 1 997. Asked by investors about where 
Edison’s savings would come from, Schmidt replied that the Edison program would deliver 
excellent results for the great majority of kids classified as special education— but in regular 
classroom settings. And while the public schools used expensive specialists and resources, the 
Edison program would deliver a program without specialists. A look at Edison special education 
data shows how the company seems to be making the policy work to its financial advantage. 

One curious feature of student demographics in the Edison schools is the relatively small 
percentage of special education students in comparison with the percentage in regular public 
schools. In 1997-98, Edison provided special education services for only 7 percent of its 
students (950 out of 13,000), while the national figure was 12 percent. In Wichita, only 5 
percent of the students were classified as special education, and in Dade County, only 4 percent. 
The Boston Charter school had a much higher percentage of special education students— 1 1 
percent (http://\vw^^'.doe.mass■edu/pic■\vw\v/prof48 1 .html) . However, this was markedly lower than the 
16 percent figure for the state and 20 percent for the city of Boston 
(http://www.doe.mass.edu/pic.www/profD35.hmil) .^ Finally, as already mentioned earlier, the 
percentage of special education students in Edison’s Worcester charter school was 14 percent, 
compared with 20 percent in the public schools. These students had relatively mild handicapping 
conditions; and a quarter of them returned to the Worcester public schools within five months. 

It is difficult to account for these differences. Public schools must either admit special 
education students living in their attendance area or pay for their education elsewhere. Is Edison 
able to screen out special education students who, they believe, could not be educated in a 
regular classroom? Is the company failing to count as special education students youngsters who 
would be so counted elsewhere? (Given the tendency of some public schools to over-use the 
special education classification,, this could be a plus for Edison.) Or are we simply looking at 
some kind of statistical fluke that will correct itself in the future? Whatever the cause, Edison is 
currently able to save money by providing services to a smaller percentage of special education 
students than regular public schools accommodate. 



’ A number of news stories and magazine articles document the special education problems Edison experienced in 
Boston. (See Education Week, February 19, 1997, The Nation, September 28, 1997, and Phi Delta Kappan, March 
1998.) Federal investigators foimd that Edison failed to notify parents of their procedural rights, failed to provide 
student support in regular and special education settings, and did not provide a full day for one of the students. 
Peggy Farber, in her Phi Delta Kappan article (March 1998) about the Boston charter, attributes the problems to a 
rocky startup. However, she does not look at the relatively small number of special education students in Boston and 
throughout the Edison schools. 
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In some cases, Edison also seems to be providing fewer services. The company indicates 
that the normal caseload for a special education teacher is 19 students. But in Wichita, 29 special 
education students (18 with learning disabilities) were handled by a paraprofessional and a 
special education administrator who split time between teaching and administering the special 
education program. In Worcester, two special education teachers served 75-80 students. And in 
Dade Coimty, according to 1996-97 school district records, one special education teacher served 
a 1,100 student school— about 44 students — and there is still only one teacher for the academic 
year of 1997-98. 

There is another way that special education students might be helping Edison's bottom 
line. Massachusetts and Michigan charter school laws split up special education money and 
allocate it to charter schools in the same way they allocate regular education money. So charter 
schools get these resources whether or not a specific charter school provides more services or 
fewer services than the sending school district (in Massachusetts) or the state average (in 
Michigan). 

But even in a district where funding is provided for specific students with specific 
disabilities, there can be room for economies. In the Wichita example cited above, 29 special 
education students, served by a teacher who doubled as a special education administrator and by 
one paraprofessional, generated about $106,000 in extra revenue in 1995-96. In 1996-97, 21 
students were served and in 1997-98, 25. Edison received $105,000 and $1 12,000, 
respectively— which, unless the special education administrator and the paraprofessional were 
handsomely paid, is about $50,000 a year more than the company spent. Admittedly, this kind of 
saving does not amount to much in an enterprise that still has only 25 schools. But this kind of 
practice will generate more significant savings if Edison attains the scale the company is hoping 
for. 



5. Skimpy implementation of Success for All. One of the big problems with any 
successful program is reproducing on a large scale the excellent results from the one or two 
original sites. The researchers who created SFA are aware of this problem, and they have 
attempted to solve it by carefully specifying the personnel and procedures necessary in any 
school that adopts SFA. Each SFA tutor should work with 9 to 1 1 students, and SFA 
recommends that, in a high-poverty school, 30 percent of first graders, 20 percent of second 
graders, and 10 percent of third graders have tutors (http://successforali.coiTi/coniponents.htnil) . 

SFA researchers have also documented the disappointing results that some schools 
implementing only part of the program have gotten. In Houston, Texas, high-implementation 
schools had positive effects while low-implementation schools had small positive to moderate 
negative effects. In addition, Slavin et al. (1997) attribute the exceptional results for low- 
performing students to full implementation of the Family Support Team, including social 
workers (see Nunnery et al., 1996, for schools in Houston and Ross, Smith, Lewis, & Nunnery, 
1996, for schools in Memphis that implemented only part of the SFA program). 

The contrast between the Edison experience with SFA in Wichita and its experience in 
Dade County seems to illustrate these findings. During the 1995-96 school year, Wichita had six 
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tutors--all of them certified teachers--which was a full complement for a school of 600 students. 
Wichita is an Edison showcase, and among the Edison schools with SFA-style evaluations, it 
performed the best. 

Edison’s Dade County school, on the other hand, was considerably short of the SFA 
complement of tutors. In order to meet SFA specifications, the 1,200-student school should have 
had at least 10 to 12 tutors just to cover grades one to three. In 1996-97, it had only five tutors 
for the entire school. (The following year, it had only 6.)* And the tutors were fi-equently used 
as substitute teachers. In 1995-96, the school also lacked the requisite SFA full-time coordinator, 
who organizes the extensive testing and regrouping of students and makes sure the program is 
being carried out according to specifications. Given the spotty implementation of SFA in Dade 
County, it should be no surprise that preliminary results showed no difference in reading 
achievement between students in the Edison school and control group students. 

The problems that have been highlighted here could result fi*om problems at individual 
schools rather than a pattern of cutting comers— it's too early to tell (though Edison's own 
comments make that unlikely with the special education examples). In any event, it is not too 
early for Edison— or its prospective clients and investors— to be alerted to possible areas of 
weakness in the project, especially since they may help explain Edison's mediocre performance 
in reading— which will be taken up in the next two sections. 



* It should be noted that Edison’s formula for allocating SFA tutors is different from that called for by SFA. 
Edison’s formula provides one tutor per 150 children. In poor schools, that means only two-thirds of the tutors SFA 
recommends (http://successforall.com/considerations.html) . In Dade County, even fewer tutors are employed. 




32 



23 



V, Success for Edison? 

The Evaluation of Edison’s SFA Program 

The Primary Reading Studies are the only full-scale evaluations that the Edison Project 
has carried out. Edison hired a distinguished researcher, Robert Mislevy, to analyze data, though 
Edison itself was primarily responsible for the data collection. The studies are a modified 
version of the standard Success for All evaluation.^ They use the same battery of standardized 
tests as an SFA evaluation and generally follow the same evaluation design. The few 
idiosyncrasies— and their effects— will be commented on in the course of this discussion. 

The Edison evaluation is very limited in scope, covering only four of the eight Edison 
elementary schools operating through the 1996-97 school year— Wichita, Mt. Clemens, Sherman, 
and Colorado Springs. Moreover, second-year evaluations are available for only two of these 
schools, Wichita and Mt. Clemens, because no control group data were collected in Sherman 
during its second year, and none are currently being collected in Colorado Springs. 

The results are tentative at best, and Edison is straightforward about this. However, the 
company is not so ready to admit that the results are extremely mediocre. More important, 
Edison never directly compares the achievement of its schools with achievement in SFA schools, 
though it sometimes makes the general claim that Edison schools are doing as well. 

There is a good reason for this reticence: None of the Edison schools, even Wichita, 
which is Edison's best, achieves at the average level of SFA schools that fully implement the 
program. And this is the case even though SFA schools typically have a higher percentage of 
students eligible for free or reduced-price lunch than Edison schools. 

Standard SFA Evaluation Design 

Every SFA school involved in a formal evaluation is matched with a control-group school 
that is similar in poverty level (percent of students qualifying for free or reduced-price lunch), 
past achievement level, ethnicity, and other factors. Schools are also matched on past 
achievement levels in district-administered standardized test scores given in kindergarten or on 
Peabody Picture Vocabulary Test (PPVT) scores given by the project in the fall of kindergarten 
or first grade. In an SFA evaluation, trained testers individually administer the following 
measures: 



^ Edison's descriptions of the Primary Reading Studies have fostered some confusion about the role of the 
Educational Testing Service in the study. Though Robert Mislevy is an employee of ETS, he was hired as an 
independent expert by the Edison Project, and the cover page of each evaluation clearly states that fact: "These 
evaluations were not carried out under the auspices of ETS, and the conclusions are solely that of the author." 
However, on its web page, Edison says that the evaluations are by ETS, and the New York Times (December 15, 
1997) repeats the error, reporting that the tests were "analyzed by the Educational Testing Service at Edison's 
behest." Needless to say, the confusion, whether intentional or inadvertent, has nothing to do with Mislevy and does 
not reflect on his competence or probity. 

The PPVT is designed to measure vocabulary in non-readers and is often used to assess school readiness for 
children in the 4 to 5 age group. Children look at pictures and match them up with words the interviewer reads out. 
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• Woodcock Reading Mastery Tests: 

1 . Word Identification assesses recognition of common sight words. 

2. Word Attack assesses phonetic synthesis skills. 

3. Passage Comprehension assesses comprehension in context. 

(Students in Spanish bilingual programs are given the Spanish versions of these 

scales.) 

• Durrell Analysis of Reading Difficulty . The Durrell Oral Reading scale is also 
administered to students in grades 1 through 3. It presents a series of graded reading 
passages that students must read aloud, followed by comprehension questions. 

A standard SFA evaluation compares raw scores on all measures using analysis of 
co-variance (a statistical technique that enables researchers to account for the independent effects 
of family background, gender, and initial ability on test scores). The evaluation design also calls 
for separate analyses for the lowest-achieving 25 percent of students in a grade. 

The Woodcock and Durrell scales are more accurate than district-administered tests and 
much more sensitive to real reading gains because they allow testers to hear children reading 
material of increasing difficulty and responding to questions about what they have read. The 
Woodcock ^d Durrell are themselves nationally standardized tests and produce norms (e.g., 
percentiles, NCEs and grade equivalents) just like any other standardized measure. 

1. Results of standard SFA evaluations. Standard SFA evaluations commonly report 
student performance in grade equivalents (adjusted for co-variance) and effect size averaged 
across individual measures (such as the Durrell and the three subscales of the Woodcock tests). 
Effect size (ES) is the difference between SFA and control-group students divided by the 
standard deviation of the control group. 

Because mathematical operations such as averaging cannot be performed on grade 
equivalents (GEs), these measures are not used in SFA statistical analyses. However, they can be 
useful for lay readers, so SFA summaries include them. Standard SFA evaluations follow 
children who began in SFA in first grade or earlier and compare them to children who attended 
the control-group school over the same period. Students who start after first grade are not 
included in evaluations because they are not considered to have received the full SFA program. 
Also, students who leave either the SFA or control-group school are dropped from the analysis. 

2. Reading outcomes in standard SFA evaluations. SFA has been in the schools for 
nearly 10 years, and numerous evaluations have been carried out to determine the program's 
effectiveness. So there is a good deal of evidence comparing the progress of SFA students with 
students in control groups. The following graph, which gathers together data for 55 schools with 
SFA evaluations, shows SFA students consistently outperforming similar youngsters in similar 
schools. The graph uses a method called "multi-site replicated experiment" and is described in 
Slavin et al., 1996a, b and Slavin and Madden, 1993. 




25 



The graph indicates the number of groups, or cohorts, of 50 to 150 students represented 
by the bar for each grade (the first grade, for example, involves 55 cohorts). The results, shown 
on the top of each pair of bars, are reported in effect size (ES). An effect size of .50 is generally 
regarded as quite successful, and, except for grade 3, where the effect size was a still 
considerable .43, the ES’s are all above .50, with the largest, 0.68, in grade 5. The numbers 
along the side of the graph represent grade equivalents (GEs) although, as already indicated, no 
calculations are based on GEs. (For more details on methods and findings, see Slavin et al., 1996 
a, b and the full site reports for the various schools the data for which are included in Slavin et 
al., 1996 a,b.) 



Figure IV- 1 

Comparison of Success for All and Control Schools in Mean 
Reading Grade Equivalents and Effect Sizes, 1988-1994. 
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Statistically significant (p=.05 or better), positive effects of SFA (compared to controls) 
were found on every measure at every grade level, from grade one through grade five. As 
already indicated, effect sizes averaged around a half standard deviation at all grade levels. 
Effects were somewhat higher than this for the Woodcock Word Attack scale in grades 1 and 2. 
But in grades 3 through 5, effect sizes were more or less equivalent on all aspects of reading. 

Effect sizes for students in the lowest-achieving 25 percent of their grades were 
particularly positive in standard SFA programs, and consistently so. They ranged from 1.03 in 
first grade to 1.68 in fourth grade. Analyses of results at individual schools found statistically 
significant differences favoring low achievers in SFA on every measure at every grade level. 

The following table gives some rough equivalents for effect sizes and normal curve 
equivalents (NCEs) and attaches them to words roughly describing the educational impact 
denoted by the effect sizes and NCEs. Both SFA evaluators and the Mislevy evaluations for 
Edison use these standards. 
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Educational 

Impact 

None 

Small 

Medium 

Large 



Effect Percent of Students Exceeding Approximate Difference in 
Size Control Group Average Normal Curve Equivalents 

(NCEs) 

.00 50% 0 

■20 58% 4-5 

.50 70% 11-12 

.80 80% 18-20 



3. Changes in effect sizes over years of implementation. Another important finding 
about SFA is that the longer the program is in place, the more the children improve. Thus, 
according to SFA research findings, the effect size for first graders, which is 0.42 at the end of 
the first program year, rises to 0.53 for first graders in the second program year, and 0.73 in the 
fourth year of the program. There is a similar pattern for second and third graders. 



Edison's Success for All Evaluation Design 

As already indicated, Edison's evaluation design follows the standard SFA evaluation 
design very closely. Some of the divergences result from Edison’s modification of the SFA 
program, others from differences in school populations, including problems Edison had or 
believed it had in constructing suitable control groups. Not all of these divergences are 
explained, acknowledged, or factored into the evaluation design, and they may or may not lead to 
more favorable overall results for Edison. However, they don't rule out a valid comparison 
between Edison's results and the SFA results just discussed. 

1. School year. Edison has a longer school year than traditional schools, so in principle, 
it tests its students in June while comparison students are tested in May. Assuming Edison is 
correct about the positive effect of an extra month on student learning, this month needs to be 
taken into account in comparing results. (Readers might want do do this by subtracting .10 of a 
grade equivalent from Edison’s test scores to account for the extra month.) 

2. Control groups 

• Mt. Clemens : Since there is only one other elementary school in the district, the entire 
control group was drawn from this school. Descriptive data from Edison indicate a close 
match of Edison students with students in the control group based on race, sex, and free or 
reduced-price lunch eligibility. Mobile students (i.e., those who enter or leave the class after 
the reading study is underway) were excluded from both Edison and control groups, in part 
because some control students transferred to the Edison school and some Edison students 
transferred to the other district school. All this is consistent with SFA evaluation design. 
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However, no measure of initial ability (such as PPVT) was ever collected from control-group 
students. So it is impossible to tell if Edison and control-group students were, in fact, well 
matched. Since no correction for differences in initial ability is possible, the outcome of the 
Mt. Clemens evaluation will always be inconclusive. 

• Wichita : The control group was drawn from all the Title I schools in the district and was 
designed to match the poverty, race, and gender of Edison students. The initial match was 
probably better than is usual in an SEA evaluation. However, the design introduced a bias in 
favor of the Edison school by counting mobile students in the control group and not doing so 
in the Edison school; mobile students should not be counted in either. 

The student population of the Wichita Edison school creates a further bias in the school’s 
favor. Many or most of the control students come from neighborhood schools. However, 
Wichita Edison is a “neighborhood magnet.” That is, it gets about half its students from 
families that choose to make the extra effort to get their child into a school outside their 
neighborhood and choose to put up with whatever inconveniences are involved in sending the 
child there. Edison could deal with the bias that results from this self-selection by analyzing 
the neighborhood and out-of-neighborhood students separately. Or it could also draw the 
control group from other Witchita neighborhood magnets (such as the pre-Edison Ingalls, 
Washington, and Kellogg) with similar poverty levels and demographics. 

• Shermaii : The control group was a combination of children whose parents volunteered to 
bring thefn in for testing and summer school students in need of remedial work. This made 
for a less-than-perfect match between Edison and control group students. However, 
variations like this are not unusual in SFA evaluations, which is why, as already mentioned, 
SFA uses analysis of co-variance (a method that statistically controls for variations in 
experimental and control groups). 

Mislevy did use statistical adjustments to report adjusted differences and adjusted effect 
size in Sherman, as SFA does in its evaluations. Edison also reduced the variations between 
the two groups by dropping ESL (English as a second language) students, who comprised 1 5 
percent of Edison students and 5 percent of control group students, from all comparisons. As 
a result of these efforts, the Sherman evaluation is valid according to SFA methodology. In 
fact, it follows SFA evaluation design more closely than the Wichita and Mt. Clemens 
evaluations, which report only unadjusted effect sizes and grade equivalents. 

Nevertheless, after making considerable and, it would seem, successful efforts to make 
the two groups in Sherman comparable, Edison said its reason for discontinuing the second 
year study was the poor match between Edison and control group students: 

The control group of students used in Sherman for purposes of comparison was not well 
matched to the Edison students because the Edison students were generally much more 
disadvantaged and the control students began the year far ahead academically. In 
controlled comparisons the two groups effectively tied in their academic progress 
(http://ww\v.aft.org/research/^edisonproiect/pages/edison-3-98/Docs/l 995Assess.html ). 

Given this reasoning, it is inconsistent for Edison to count Mt. Clemens as a success. As 
already noted, no data on initial ability were ever collected there, so the comparability of 
Edison students and control group students cannot be established. 
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Of course, if there were a problem with the Sherman control group, Edison had a chance 
to improve the match in the second year evaluation. ' ' Instead, no control group data were 
collected, which effectively cancelled the evaluation. Since this decision coincided with 
Sherman's second year of disappointing scores on the Texas state assessment (TAAS), it 
raises some questions about Edison's willingness to risk evaluating a product that might not 
be up to the expectations it advertises. 

• Colorado Springs: The Primary Reading Study here faced a problem similar to Sherman's: 
The control group tested higher on the pre-test than the Edison students. The reading study 
did adjust for the pre-test difference. However, it failed to adjust for race, gender, and free- 
lunch status, even though such adjustments are part of SFA evaluation design and were made 
in other Edison schools. According to school officials, race and free-lunch data were 
available and could still be added to the study. But given the current gaps in the data, any 
comparisons between SFA achievement nationally and achievement in Colorado Springs 
would be problematic. And since Edison has not made arrangements to collect control group 
data for the second year, the company has effectively terminated the second year of the 
evaluation. Again, one might ask if these problems result from ineptitude or from Edison's 
wish to shine the spotlight away from mediocre first-year results at an Edison school. 

3. Student populations. Edison schools eiuoll more poor students than the average 
American school, but the students are not as poor as the company says. The 1995-96 and 1996- 
97 Wichita evaluation claims that: 

...the students in the [Edison school] and control groups are generally disadvantaged; 
about three-fourths of them participate in the free or reduced lunch programs. On the 
whole, they are probably more like the lower 25th percentile of the students in the Slavin 
et al. sites than the typical students in those sites (p. 7). 

In fact, except for Dade County, all of the Edison schools opened in 1995-96 and 1996-97 are 
less disadvantaged than the typical SFA school, where free-lunch students usually comprise at 
least 90 percent of enrollment (Slavin, Madden, and Wasik, 1996). The percentage of students 
qualifying for free or reduced-price lunch at the Edison school in Wichita fell to 69 percent in 
1997-98, and the school is in danger of losing its funding as a Title I school-wide project. The 
student populations at Sherman (64 percent eligible for free or reduced-price lunch), Boston (5 1 
percent), and Mt. Clemens (48 percent eligible) are more advantaged than the population at 
Wichita. Moreover, the percentages of students eligible for free or reduced-price lunch have 
been dropping at Edison schools. In addition to the decline already noted in Wichita, the 
percentage of eligible students fell from 54 to 44 percent in Colorado Springs over a two-year 
period, from 59 to 48 percent in Mt. Clemens, and from 69 percent to 51 percent in Boston. 
However, the percentage of free or reduced-price lunch in Dade County rose from 86 to 90 
percent. 



In Mt. Clemens in the year-two evaluation, Edison and control-group students were “rematched” according to 
ethnicity, gender, and eligibility for free or reduced-price lunch. Since there were no data about students’ initial 
ability, this rematching was futile. 
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VI. Results of the Edison Primary Reading Studies 

The results of the Edison Primary Reading Studies show the performance of Edison 
schools to be unremarkable thus far. However, Edison's news releases and promotional materials 
do not necessarily reflect what the data show. In December 1997, Edison made this claim: 

Primary reading achievement studies show that students who began at an Edison school 
in kindergarten or first grade are consistently developing stronger skills than similar 
students locally. At [the Wichita and Mt. Clemens elementary schools], students in 
grades K-2 are strongly outperforming their counterparts in local schools in reading; if 
the trends continue, Edison students could be a full-year ahead of similar students by the 
end of fifth grade (Edison Press Release, Dec. 15, 1997. Also, see 
http ://w vvw.edisonproj ect. com/annualframe .html ) . 

And a past posting on the Edison Project’s Web site states that “at the end of year one, primary 
reading achievement is slightly above nationally normed grade levels in kindergarten, on grade 
level in first grade, and slightly below grade level in second grade” 

(littp://\vww.aft.org/research/edisonproiect/pa,ges/edison-3-98/Docs/1995Assess.hrml ). 

The statement fi'om the press release is simply not true. It was apparently derived by 
extrapolating the record of SEA schools (see table, p.26) to the Edison schools rather than fi’om 
data contained in the Edison Primary Reading Studies, and it is contradicted by Edison's own 
record. The statement posted on the Web, while less hyperbolic, is also untrue. Edison has been 
less successful than the average SFA school. 

It would be more accurate to say that Edison's performance is uneven. While the 
favorable comments cited above apply to Wichita, students in Edison's Sherman, Texas, school 
performed below grade level in all grades. In Mt. Clemens and Colorado Springs, kindergarten 
students performed at grade level, not above it. In grade one, only Wichita students performed at 
grade level in the year-two evaluation. 

Table VI-1 allows a comparison of SFA national data and data fi’om the Edison Primary 
Reading Studies. The SFA data, which are given in the first section of the table, are taken fi’om 
the most recent compilation of SFA evaluations, comprising several dozen schools (Slavin et al, 
1996a, b). The Edison data for the six comparable Edison reading evaluations appear below the 
SFA data. Because SFA evaluations do not report kindergarten results, comparability between 
Edison and SFA does not properly begin until grade one. 
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Table VI-1 

Reading Achievement of Edison Project and Success for All Schools 



K 



Grade 1 



Grade 2 



Grade 3 



Grade 4 Grade 5 



Success for All Evaluations-AII Years 



Number of schools 


55 


36 


33 


13 


6 


Number of students 


5,000+ 


3,600+ 


3,300+ 


1 ,300+ 


600+ 


Adjusted Effect Size 


0.53 


0.50 


0.43 


0.53 


0.68 


Significant at .05 level 


Yes a 


Yes a 


Yes a 


Yes a 


Yes a 


GE-Successfor All 


1.8 


2.6 


3.3 


3.9 


4.8 


G E-Control 


1.6 


2.2 


2.7 


3 


3.8 





Pupils 


Pupils 


Pupils 


Pupils 




Mt. Clemens-Year 1 




















Unadjusted Effect Size 


0.48 




0.36 




-0.15 










Subtest sig at .05 level 


3 of 4 




1 of4 




0 of 4 










Average GE-Edison (b) 


1.0 


96 


1.8 


81 


2.1 


75 








Average GE-Control (b) 
Mt. Clemens-Year 2 


0.8 


61 


1.6 


60 


2.2 


60 








Unadjusted Effect Size 






0.26 




0.45 




-0.24 






Subtest sig at .05 level 






Oof 4 




1 0f4 




Oof 4 






Average GE-Edison (b) 






1.9 


35 


2.7 


42 


2.9 


37 




Average GE-Control (b) 






1.7 


35 


2.2 


42 


3.1 


37 




Wichita-Year 1 




















Unadjusted, Effect Size 


0.68 




0.37 




0.07 










Subtest sig at .05 level 


4 of 4 




2 of 4 




0 of 4 










Average GE-Edison (b) 


1.3 


94 


2.0 


96 


2.7 


88 








Average GE-Control (b) 


1.1 


83 


1.8 


88 


2.6 


89 








Wichita-Year2 




















Unadjusted Effect Size 






0.49 




0.23 




-0.04 






Subtest sig at .05 level 






4 of 4 




0 of 4 




Oof 4 






Average GE-Edison (b) 






2.1 


77 


2.8 


81 


3.4 


75 




Average GE-ControI (b) 






1.8 


51 


2.6 


58 


3.4 


52 




Sherman-Year 1 




















Adjusted Effect Size 


-0.18 




-0.03 




0.02 










Subtest sig at .05 level 


1 of 4 




1 0f4 




0 of 4 










Average GE-Edison (b) 


1.0 


103 


1.6 


77 


2.4 


77 








Average GE-ControI (b) 


1.2 


87 


2.0 


80 


2.7 


62 








Colorado Springs-Year 1 




















Adjusted Effect Size 


0.20 




-0.09 




-0.22 










Subtest sig at .05 level 


2 of 4 




2 of 4 




1 of4 










Average GE-Edison (b) 


1.0 


100 


1.8 


125 


2.4 


99 








Average GE-Control (b) 


1.0 


105 


2.1 


103 


2.8 


86 









Source: Slavin, Madden & Wasik (1996); Edison Primary Reading Studies by Robert Mislevy. 

a. Statistically significant for every measure at all grade levels favoring SFA. 

b. Average of GE for each of the four subtests (for illustrative purposes— mathematical operations 

should not be performed on GEs). 
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The performance of Edison schools was similar to the SFA average in grade one of the 
Wichita year-two evaluation and grade two of the Mt. Clemens year-two evaluation (although 
only one of four tests was statistically significant). In any case, since the Mt. Clemens study 
contained no control for initial ability, results there should be viewed cautiously, and perhaps as 
inconclusive. Sherman and Colorado Springs showed statistically insignificant effects favoring 
the control group. However, defects in the implementation of the evaluation design in Colorado 
Springs makes these results inconclusive. The other four valid comparisons were in the .26 to 
.37 range (for the most part, with statistically insignificant differences for at least three of the 
four individual reading tests), which is below the SFA average. 

Since SFA evaluations include only those students beginning SFA in the first grade or 
earlier, the second-grade Edison results in year one and the third-grade results in year two 
evaluations should not be compared to SFA results. The comparison of effect sizes for SFA and 
the comparable Edison studies are graphed in Figure VI-1 . Of eight valid comparisons, two 
would be considered moderate effects (equivalent to 10 to 11 NCEs) favoring Edison, four would 
be considered small effects favoring Edison (4 or 5 NCEs), and two (grade 1 in Sherman and 
Colorado Springs) reveal no educational effect.'^ The Sherman evaluation, which would have 
yielded a second-grade and another first- grade comparison, was discontinued after the first year. 
It should be noted that SFA researchers do not drop imsuccessfiil schools from a study; doing so 
would create a serious bias in any results. If the second-year study in Colorado Springs is not 
conducted (and the first-year not completed), no serious conclusions about the effectiveness of 
Edison’s SFA program in these schools will be possible because they will be have to be based 
only on the remaining studies in Wichita and Mt. Clemens. And the Mt. Clemens findings are 
questionable, given the general lack of statistical significance and the failure to control for initial 
ability. 

Interpreting the Results 

Here are some caveats for interpreting the results of the Primary Reading Studies and 
Edison's claims about its progress. 

1. Looking at non-SFA effects. The Primary Reading Studies follow SFA 
protocol in not including students who did not begin SFA in grade one (though effect sizes for 
these classes are noted). Nevertheless, negative or small effect sizes for these classes should not 
be ignored. The Edison program includes many components besides SFA, such as longer school 
days and years and high spending on educational technology. Small or negative effect sizes 
strongly suggest that these attractive “extras” are, so far, not successful in improving student 
achievement. 



A moderate effect (ES of .50) means that about 70 percent of the higher scoring group does better than the 
average of the lower group. A small effect size (ES of .20) means that 58 percent of the higher group does better 
than the average of the lower group. 
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Figure VM -Comparison of Success 
For All and Edison Primary Reading Studies 




Success Wichita Wichita Mt. Mt. Sherman Colorado 

For All Year 1 Year 2 Clemens Clemens Year 1 Springs 
Average Year 1 Year 2 Year 1 



O Grade 1 
■Grade 2 



2. An off-the-wall calculation. In the 1997 annual report, Edison says that Colorado 
Springs students “outgained” the control group, with effect sizes ranging from an excellent .70 in 
kindergarten to a small, but still significant, .20 in second grade. Edison got these numbers by 
calculating the difference between the pre-test (the PPVT, a non-verbal test of intelligence) and 
the average of four post-tests (which assess verbal skills such as word identification and passage 
comprehension) and then rendering the results of this calculation in effect sizes. This strategy 
ignored Mislevy's approach to the problem— which is also the one used by SEA researchers. 
Mislevy statistically adjusted each of the four post-tests for the effects of the pretest using 
analysis of co-variance. Mislevy’s results showed that the average effect size in kindergarten 
improved from .06 to .20; in first grade from -.36 to -.09; and in second grade from -.48 to -.22.— 
in other words, they were less impressive than the ones Edison got with its off-the-wall 
calculations. However, since Mislevy’s results are not adjusted for race, gender, and eligibility 
for free or reduced-price lunch, they are also not conclusive, either. 

3. Subtest effect sizes. Table VI-2 presents data for individual tests, including statistical 
significance. The word-attack effect size for grade 1 in the first-year at Mt. Clemens (.84) was 
about four times larger than any of the other three tests and raised the combined effect size of all 
tests to .36— still a below-average score for SFA programs. Strong word-attack scores are 
common in the early grades in SFA schools. In Mt. Clemens, no more than one in four 
individual tests was statistically significant in grades 1, 2 or 3. 



See page 24 for a description of these tests. 
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Table VI-2 

Subtest Effect Sizes in Edison Primary Reading Studies 



Woodcock 





Durrell 

Oral 


Word 

ID 


Word 

Attack 


Passage 

Comp. 


Average 


Mt. Clemens-Year 1 


Kindergarten 


0.50 


0.55 * 


0.54 * 


0.34 * 


0.48 


Grade 1 


0.27 


0.17 


0.84 * 


0.16 


0.36 


Grade 2 


-0.25 ** 


-0.18 


-0.02 


-0.19 


-0.15 


Mt. Clemens-Year 2 


Grade 1 


0.42 


0.25 


0.34 ** 


0.11 


0.28 


Grade 2 


0.55 * 


0.42 


0.45 


0.36 


0.43 


Grade 3 


-0.08 


-0.21 


-0.32 


-0.17 


-0.24 


Wichita-Year 1 


Kindergarten 


0.71 * 


0.60 * 


0.88 * 


0.53 * 


0.68 


Grade 1 


0.43 * 


0.29 


0.41 * 


0.34 


0.37 


Grade 2 


0.11 


0.03 


0.11 


0.03 


0.07 


Wichita-Year 2 


Grade 1 


0.44 * 


0.57 * 


0.41 ** 


0.52 * 


0.49 


Grade 2 


0.30 


0.26 


0.14 


0.21 


0.23 


Grade 3 


-0.02 


-0.12 


-0.10 


0.03 


-0.04 


Sherman-Year 1 


Kindergarten 


-0.12 


-0.25 ** 


-0.20 


-0.14 


-0.18 


Grade 1 


0.03 


-0.14 ** 


0.11 


-0.10 


-0.03 


Grade 2 


0.18 ** 


-0.10 


-0.02 


0.03 


0.02 


Colorado Springs-Year 1 


Kindergarten 


0.05 


0.17 


0.29 * 


0.30 * 


0.20 


Grade 1 


-0.27 * 


-0.27 * 


0.21 ** 


-0.04 


-0.09 


Grade 2 


-0.16 


-0.37 * 


-0.05 


-0.29 ** 


-0.22 



* Statistically significant with a 5 percent chance of error. 
** Statistically significant with 1 0 percent chance of error. 



Edison claims that “statistical significance” is the basis for deciding success in the 40 
grade- level comparisons featured in their Annual Report on School Performance (1977). The 
data presented in Table VI-2, however, show that Edison does not necessarily live up to its own 
standard. The term “statistical significance” refers to the numeric test used by researchers to 
determine the probability that individual findings are different from chance. In the Primary 
Reading Studies, Edison counted the following grade-level comparisons as successes when they 
were, in fact, statistically insignificant— that is, at least three of four subtests were statistically 
insignificant: 
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• Wichita (1997) second grade 

• Mt. Clemens (1996) first grade; (1997) first grade and second grade 

• Colorado Springs (1996) first grade and second grade 

In Sherman , Edison, surprisingly, counted second grade as a “no success” when, in fact, there 
was no statistical significance to the finding. In all, only 5 grade-level comparisons in any 
Edison schools showed statistical significance in at least two of Xhtfour subtests. 

4. Changes in effect sizes over years of implementation. SFA claims that its program 
becomes more effective with each successive year of implementation (see the first panel of Table 
VI-3). This means that an incoming first grade does better in the second year SFA is in place 
than the first grade during year-one of the SFA program in that school, etc. There is not yet 
enough data to establish whether or not this pattern holds for Edison schools. As Table VI-3 
shows, grade one scores in Mt. Clemens went up during the second year, and they went down in 
Wichita. Since no other Edison schools are currently collecting multi-year data for control-group 
schools, in the forseeable future, only Wichita and Mt. Clemens will allow us to reach 
conclusions about changes in effect size over the years. 



Table VI-3 

Trends in Effect Size by Year of Program Implementation 









Years Proqram InriDlementation 








Year1 


Year 2 


Years 


Year 4 


SFA Schools 












Grade 1 


c 


0.49 


0 . 53 I 


0.58 


0.73 


Grade 2 






* 0.41 


0.47 


0.87 


Grade 3 






★ * 


0.32 


0.54 


Edison-Mt. Clemens 












Kindergarten 




0.48 








Grade 1 


L 


0.36 


0.26I 






Grade 2 




-0.15 


* 0.45 






Grade 3 






-0.24 * 






Edison-Wi Chita 












Kindergarten 




0.68 








Grade 1 


L 


0.37 


0.491 






Grade 2 




0.07 


* 0.23 






Grade 3 






-0.04 * 







* Not counted in SFA evaluations because students did not begin program in grade 1 . 
Source: Slavin, Madden & Wasik (1996); Edison Primary Reading Studies by Robert Mislevy. 
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Figure VI-2 Comparison of Success For All and Dodge-Edison 
Effect Sizes for Lowest Achieving 25 Percent 



0 ) 

N 

in 

u 

£ 

Ui 




SFA Gr. 1 SFA Gr. 2 Wichita Gr. 1 Wichita Gr. 1 Wichita Gr. 2 

95-96 96-97 96-97 



5. Low-performing students. SFA is especially effective in working with students in 
the bottom 25 percent of achievement, as can be seen in Table VI-4. Although Edison does not 
make this claim about its SFA program specifically, the company does say that it is having this 
kind of success with very low achievers in Wichita and elsewhere. For example, a recent Time 
article (October 27, 1997) on Edison’s elementary school in Wichita asserts that “. . .teachers 
have cut in half the gap in test scores between poor and affluent children. The school made 
similar headway in demolishing the difference between minority and white students’ 
performance.” Recently, Edison asserted that, “Students at every level (from the highest quartile 
to the lowest) are making strong academic progress. Dodge-Edison [Edison’s Wichita 
elementary school] provides the longest-term illustration, though every school shows similar 
patterns” (Edison press release, Dec. 15, 1997). 

However, the data paint an entirely different picture about the achievement of Wichita’s 
lowest 25 percent. The results are uniformly poor, as shown in Figure VI-2 and Table VI-4. The 
lowest-achieving students perform substantially below the SFA average and also below the 
average Edison student. Edison did not do a low-performing student analysis in any of its other 
schools, but since the company already has the necessary data, it could set the record straight— 
and reassure school districts it is courting— by performing these analyses. 
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Table Vi-4 

Effect of the Edison Project on Low-Performing Students 





K 


Grade 1 


Grade 2 


Grade 3 


Success for Ail Evaluation 










Number of schools 




15 


15 


13 


Number of groups 




37 


21 


13 


All SFA Students 










Adiusted Effect Size 




0.58 


0.65 


0.53 


Subtests Sig. at .05 level 




4 of 4 


4 of 4 


4 of 4 


G E-Success for All 




1.9 


2.7 


3.5 


GE-Control 




1.7 


2.2 


2.8 


Students in Lowest 25% on Pretest 








Adiusted Effect Size 




1.04 


1.47 


1.49 


Subtests Sig. at .05 level 




4 of 4 


4 of 4 


4 of 4 


GE-Success for All 




1.6 


2.0 


2.7 


GE-Control 




1.3 


1.6 


1.9 


Edison -Wichita, 1996-Students in Lowest 25 % 






Edison Students 


23 


22 


22 




Control Students 


24 


25 


22 




' Unadjusted Effect Size 


1.10 


0.23 


-0.21 




Subtests Sig. at .05 level 


1 of4 


Oof 4 


Oof 4 




G E-Edison 


1.0 


1.8 


2.1 




GE-ControI 


0.8 


1.6 


2.2 




Edison-Wichita, 1997-Students in Lowest 25 % 






Edison Students 




21 


22 


15 


Control Students 




19 


17 


15 


Unadjusted Effect Size 




-0.06 


0.21 


-0.22 


Subtests Sig. at .05 level 




0 of 4 


Oof 4 


0 of 4 


GE-Edison 




1.6 


2.5 


2.6 


GE-Control 




1.6 


2.2 


2.8 



Source: Slavin, Madden & Wasik (1996); Edison Primary Reading Studies by 



Robert Mislevy. 



O 

ERIC 



46 



VIL School-By-School Analysis of 
Student Achievement in Grades 3 to 5 

Edison Project's Assessment Design 

Edison's description of how it plans to assess the progress of students in its schools is 
straightforward and follows generally accepted procedures. In practice, however, the company 
sometimes presents its data in ways that tend to exaggerate test scores, and it often ignores 
comparisons to other schools, students, tests, and years that would render results less impressive. 
Here is how the company describes its assessment design: 

[D]uring the 1995-96 school year, the Edison Project's first four schools administered a 
variety of standardized tests and assessments to measure student achievement. These 
tests generally were administered in the fall and early winter and serve as baselines for 
evaluating student progress in the schools' second and subsequent years of operation. 

The initial scores show that partnership school students are beginning their Edison 
experience at or below national normed grade levels and state standards— not uncommon 
for schools such as these that serve predominantly disadvantaged populations. Testing 
in subsequent years will reveal how much students gained on their starting levels 
(http://www. aft.org/research/edisonproiect/pages/edison-. "-9S'Docs/1995.'\ssess.html ). 

In three of the first four Edison schools, the company adheres pretty closely to the 
procedures described here. Wichita and Boston students both took MAT/7 assessments, and Mt. 
Clemens students took the Michigan state assessment (MEAP) and the Iowa Test of Basic Skills 
(ITBS). In all three schools, the baselines were slightly lower than would ordinarily be the case. 
Students were taking the MAT/7 or ITBS for the first time, and test scores generally slump when 
a standardized test is first administered because students and teachers are unfamiliar with it. But 
the low baseline is not a handicap when it comes to demonstrating impressive progress. Quite 
the contrary— when students get used to a new test, the school can take credit for the jump in 
scores, even though some of the improvement would have occurred regardless of the program in 
place. This is a problem in any public school that switches to a new test. 

Most of the other schools that opened in 1 996-97 will not be able to make this important 
before-and-after comparison because they did not collect fall baseline data (as shown in Table 
VII- 1). And there are some problems with baseline data in the Sherman, Texas, and Colorado 
Springs schools, which suggest that Edison is not averse to picking and choosing among 
baselines. 

Sherman has no fall baseline data for the year it became an Edison school. There are, 
however, good data from the Texas state assessments (TAAS) administered the previous spring, 
and, since the Edison school essentially serves the same neighborhood and students, it would be 
perfectly legitimate to use these data to establish the baseline. Edison, however, preferred to do 
without baseline data, perhaps because the TAAS reading scores from the spring before Edison 
took over were approximately 20 points higher in grade 3 reading and grade 4 math than after the 
first year with Edison. 
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The same spring data that Edison found unacceptable for Sherman passed muster in 
Colorado Springs. This school district tests twice a year, measuring student progress from fall to 
spring. But Edison chose to use the spring ’95 data for the Colorado Springs baseline, which had 
the effect of making Colorado Springs results look better than they otherwise would. 

Table VII- 1 shows the tests administered to Edison elementary schools open in 1995-96 
and 1996-97 by the state, the district, and Edison; the grades for which we have data; the date of 
the baseline; and the dates of the assessments administered thus far. The column designated as 
baseline is either a spring test the year before Edison took over or a test in the fall of Edison’s 
first year. 



When fall testing data from Edison’s second year (1997) became available, Edison began comparing fall 1996 to 
fall 1997 and dropped the use of the pre-Edison spring testing data (Annual Report on School Performance, 1997 
http://\\'W'w. edisonproiect.com/aimualframe.html) . This fall-to-fall comparison suffers from the same problems, 
identified in this report (see page 69), as the spring-to-spring comparisons. Since the fall 1997 test is used by the 
district as a baseline for the spring test, the AFT will not present the fall testing data imtil spring test scores are 
available. Edison also administers a second standardized test (the ITBS) in December. The changes in scores 
between December 1996 and December 1997 are similar to the fall-to-fall changes in the district’s standardized 
tests. The ITBS scores will be incorporated into this analysis when the school district spring test results become 
available. 
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Table VIM 

District and State Assessment for Edison Elementary Schools 



Edison Assessments 





Grades 


Baseline 




First 


Second 


Third 


School Opened Fall 1995 
Wichita 


Kansas State Assessments 


3,4 


Apr 95 


a 


Apr 96 


Apr 97 


Apr 98 


District tests (MAT/7) 


3,4,5 


Oct-95 




Oct-96 


Oct-97 


Oct-98 


Mount Clemens 


State tests (MEAP) 


4 


Oct-95 




Oct-96 


Jan-98 


Jan-99 


Edison tests (ITBS) 


3,4,5 


Dec-95 


b 


Dec-96 


Dec-97 


Dec-98 


Sherman 


State tests (TAAS) 


3,4 


May-95 


c 


May-96 


May-97 


May-98 


Boston (Charter School) 


State tests (MEAP) 
State tests (ITBS) 


4 

3 


none 

none 


d 


Apr-96 


e 

Apr-97 


e 

Apr-98 


Edison tests (Stanford) 


3,4,5 


none 




Apr-96 


Spring 97 


Spring 98 


Edison tests (MAT/7) 


3,4.5 


Oct-95 




Oct-96 (q) 


Oct-97 


Oct-98 


School Opened Fall 1996 
Colorado Springs 


District tests (DALT) 


3,4,5 


Fail 95 to 
Spring 96 


f 


Faii 96 to 
Spring 97 


Fall 97 to 
Spring 98 


Fall 98 to 
Spring 99 


Dade County 


District tests (DALT) 


1.2,3,4,5 


Jun-96 


d 


Jun-97 


Jun-98 


Jun-00 


Lansing (Charter School) 


State tests (MEAP) 


4 


Oct-96 




Jan-98 


Jan-99 


Jan-99 


Edison tests (MAT/7) 


3,4 


none 




Spring 97 


Spring 98 


Spring 99 


Worcester (Charter School) 


State tests (ITBS) 


3 


none 




Apr-97 


Apr-98 


Apr-99 


Edison tests (Stanford) 


3 


none 




Jun-97 


Spring 98 


Spring 99 


Edison tests (Stanford) 


4 


none 




Apr-97 


Spring 98 


Spring 99 


Edison tests (MAT/7) 


5,6,7 


none 




Jun-97 


Spring 98 


Spring 99 



a. Edison added about 50% to 1994-95 enrollment from out-of-neighborhood students. 
In both years, Dodge had identical racial and economic makeups. 

b. Detailed data from test publisher not available to AFT. 

c. As in Mt. Clemens, the Sherman school is a neighborhood school. 

d. Baseline is pre-Edison scores of individual students. Study includes control group. 

e. State administration of MEAP suspended after 1996; replaced with Stanford. 

f. District measures progress from fall to spring, but spring to spring is also possible. 

g. Tested in October, but used test form and spring national norms from previous grade. 
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Filling in the Outline 



As already noted, there are certain problems connected with Edison’s handling and 
presentation of test data. Some will be pointed out in discussions of individual schools, but a 
few general observations follow. 

1. Percentile rankings overstate Edison's gains. Using national percentile rankings to 
measure student achievement can be misleading. Percentiles are rankings, not measures of 
knowledge; they can tell if one student scored better than another but not how much better. 
Moreover, rankings on standardized tests are especially unreliable around the SO'** percentile, 
where a large number of students with almost the same score are bunched together in the middle 
of the bell-shaped curve that is characteristic of standardized tests. As a result, small changes in 
knowledge create big changes in percentile rankings. In many of Edison’s early schools (e.g., 
Wichita, Mt. Clemens, and Boston), scores tend to improve from just below average to just 
above average, which means that all of the problems associated with percentiles come into play. 

Using normal curve equivalents (NCEs) avoids these problems. They are normalized 
national percentile rankings, which are similar to percentiles--the mean is 50 with a range of 1 to 
99. However, NCEs represent equivalent gains in knowledge, not rankings. A student would 
have to have the same number of correct answers to move from 35 NCEs to 45 NCES as he 
would to move from 45 NCEs to 55 NCEs, which is not true of percentile rankings. Because of 
this problem with percentiles, the federal government now allows only NCEs in Title I 
evaluations. ' 

The MAT/7 scores in Wichita are a good illustration of the problem with percentiles. 
Wichita is the Edison school with the biggest achievement gains, and many students have moved 
from below average to above average (right in the middle of the bell-shaped curve). If you 
follow the same students from grade 3 to grade 5 over a two-year period, the change in the 
reading score is 12 NCEs (which is generally considered good) as opposed to the more 
impressive seeming 23 percentiles (see panel A in Table VII-2). The change in math is 18 NCEs 
(also good) rather than 32 percentiles, which sounds much better. Comparing grade 5 reading in 
1995 to grade 5 reading in 1997 shows a gain of 4 NCEs compared to 8 percentiles (panel B in 
Table VII-2). In math, fifth graders improved by 12 NCEs, about half of the 23 percentile 
change. The NCEs rankings convey student results much more precisely; the percentiles can 
make the results sound more impressive. 
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Table VII-2 

Difference Between Percentiles and NCEs On Wichita's MAT/7 

Reading Math 





%tiies 


NCEs 


%tiles 


NCEs 




A. Diagonal Comparison 






Grade 3, 1995 


39 


44 


39 


44 


To Grade 4, 1996 


49 


50 


52 


51 


To Grade 5, 1997 


62 


56 


71 


62 


One- Year Change 


10 


6 


13 


7 


Two-Year Change 


23 


12 


32 


18 


Grade 4, 1995 


47 


48 


37 


43 


To Grade 5, 1996 


60 


55 


65 


58 


One-Year Change 


13 


7 


28 


15 




B. Conventional Comparison 




Grade 4, 1995 


47 


48 


37 


43 


Grade 4, 1996 


49 


50 


52 


51 


Grade 4, 1997 


53 


52 


63 


57 


Two-Year Change 


6 


4 


26 


14 


Grade 5, 1995 


54 


52 


48 


50 


Grade 5, 1996 


60 


55 


65 


58 


Grade 5, 1997 


62 


56 


71 


62 


Two-Year Change 


8 


4 


23 


12 



Source: Wichita Pubiic Schools. 
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Figure VIM MAT/7 Math and Reading Results for 
Current Fourth and Fifth Graders 
Using NCEs, With Scale Beginning at Zero 




Reading Math 

Third to Fourth Grade 



Reading Math 

Fourth to Fifth Grade 



Exhibit VII 

In this handout used by 
Edison, the graphing of 
percentile rankings 
exaggerates achievement 
growth, which is more 
accurately portrayed in 
Figure VII- 1. Starting the 
axis at 20, rather than 0 
also exaggerates 
achievement growth. For 
example, grade 5 math 
scores in 1996 appear to be 
nearly three times larger, 
when the change from 35 
to 64 percentiles does not 
even represent a doubling 
of the percentile rank. 



Results: Wichita KS 



Percentile Scores 1995 vs, 1996 on MAT-7 
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2. Comparing student achievement across grade levels provides incomplete data. 

There are at least three ways to compare progress in the average academic achievement of groups 
of students over time: 

• Compare the same grade in different years (e.g., all third graders in one year to all third 
graders the previous year). 

• Compare the same cohort of students in different grades (e.g., all fourth graders in one year to 
all third graders in the previous year). 

• Compare the same students in different grades (e.g., only fourth graders who also had third 
grade scores, ignoring students without both scores). 

The first two ways of comparing student achievement are just different ways of looking at 
the same data. Almost all reports of assessment results from states or school districts use the first 
method— comparing the same grade in different years— which provides information on how well a 
particular school is doing at a particular grade level over time. One problem with this method is 
that every year of data contains a completely different group of students. 

The second method of comparison, which is the one Edison prefers, follows the same 
cohort of students from grade to grade. But because students change schools all the time, this 
method also compares different groups of students. So in both cases, changes in student 
achievement reflect not only the impact of the school but also the impact, if any, of a changing 
student body. 

The cohort method has the additional problem of providing fewer achievement 
comparisons because one grade is always the baseline grade. For example, if students are tested 
in grades 3,4 and 5, we know only what happens as students move from grades 3 to 4 and grades 
4 to 5, but nothing about grade 3. Analyzing trends over a period of more than one year also 
becomes very problematic and negates most of the advantages of comparing students as they 
move from grade to grade. For example, if students moving from grades 3 to 4 in one year are 
compared to students moving from grade 3 to 4 in the previous year, you are comparing two 
completely different groups of students, just as you do with the first method. 

However, if student achievement data are presented for all years and all grade levels, then 
the reader is free compare data both ways. Tabular data in this report are always presented in this 
way. Edison testing data, however, are usually limited only to the second method, forcing 
readers to look at the data in only one way. Panel A of Table VII-2 replicates Edison's preferred 
way of showing student achievement data: by comparing grade 5 test results to grade 4 results 
from the previous year. Panel B of Table VII-2, which compares fourth grade scores and fifth 
grade scores in three different years, indicates less progress in student achievement. 

Comparing only those students with test scores in two consecutive years (i.e., testing 
exactly the same students over time), which is the third comparison method bulleted above, has a 
lot of intuitive appeal. When Edison compares student achievement across grade levels, it has 
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been very unclear about whether or not it is limiting the comparison to exactly the same students 
or to the same cohort of students as it progresses from one grade to the next. Based on test 
publisher's data analyzed by the AFT, it appears that Edison generally compares the same cohort 
of students across grade levels rather than exactly the same students.*^ 

3. When there are no control schools, the context for interpreting test results is 
inadequate. Although Edison's Primary Reading Studies assessed student progress in relation to 
a control group of similar students, Edison generally compares results for its students only to 
national norms on norm-referenced tests and to state averages on state assessments. The problem 
with this is that states and most school districts always find a way to do better than the national 
average. (This is the so-called Lake Wobegon effect, named after the mythical town “where all 
students are above average.”) So careful comparisons also look at achievement patterns in 
similar schools, which form control groups. The University of Maryland-Baltimore County 
(1995) evaluation of Education Alternatives, Inc. (EAI), in Baltimore revealed the importance of 
comparison schools. On surveys of teachers, parents and students, EAI looked as though it was 
doing superior work, but the survey results for a set of demographically similar control schools 
showed that satisfaction with EAI was no better than satisfaction in the control schools. 

One exception to Edison’s general practice in this regard is Mt. Clemens. There are only 
two elementary schools in the district, and Edison makes comparisons to the non-Edison school 
and the state average. However, outside of the Primary Reading Studies, Edison provides no 
data on the comparability of the two schools with respect to student characteristics. Also, Edison 
compares the grade 3 state reading assessments of its two charter schools in Massachusetts to 
school district averages in Boston, Chelsea, Somerville, Lowell, Brockton, and Worcester. In 
Colorado Springs, Edison compares its school to the district average but not to other similar 
schools. 

4. “Neighborhood magnet” data, unless analyzed separately, skew results. The 
Edison schools in Wichita, Mount Clemens, Sherman, and Colorado Springs are partly magnet 
schools. Dade County is one of three neighborhood elementary schools residents can choose, but 
it is not a district-wide magnet. The other Edison schools are charter schools. As already noted 
in connection with the Primary Reading Studies (see p. 14), admission by lottery may eliminate 
school selectivity, but, as studies of magnet schools have shown, it does not eliminate self- 
selection by parents who choose to apply. Parents who make the effort to get their child into a 
specific school outside of their neighborhood are likely to be more involved with their child's 
education and school. 

Experiences in Wichita, Sherman, Colorado Springs, and Mt. Clemens also suggest that 
self-selection tends to decrease the percentage of students eligible for free or reduced-price lunch. 
In Mt. Clemens, the percentage dropped from 59 to 48 percent, and in Wichita it has declined 



** Despite its intuitive appeal, comparing exactly the same students is a very problematic procedure. National norms 
are based on a representative sample of students, not a select group of students who stayed at the same school for at 
least two years. Precisely because students who leave and enter the school are eliminated from the test-score 
comparison, the use of national norms (as a con^rarison measure) is inappropriate. Another problem is that when 
some students who do not do well leave the school, their absence the following year is likely to contribute to higher 
average test scores because those unsuccessful students are eliminated from the analysis. 



O 

ERIC 



54 



45 



over two years from 76 to 64 percent.'® This, in turn, is likely to produce a rise in test scores. 

The remainder of Section VII examines student achievement in each of the first four 
Edison schools. The section also assesses early evidence in Dade County and Colorado Springs. 
Testing data come from the Edison Project, school district officials, and state governments. 
Generally, testing data provided by Edison to the AFT were backed up with detailed reports from 
test publishers. 



See page 29 for more details about decreases in the number of students eligible for free and reduced-price lunch 
in Edison schools. 

In 1995-96, it was reported on the Massachusetts Department of Education Web page that nearly 40 percent of the 
Boston charter school students came from private schools, which is obviously another measure of self-selection. 
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A. Dodge-Edison in Wichita 

Everybody likes a success story— and the more dramatic the better. So it should be no 
surprise that articles about Edison's elementary school in Wichita (and Edison's own promotional 
materials) tend to exaggerate both the sorry state of the school before Edison took over and the 
degree to which Edison has turned it around. 

An article in Time (Oct. 27, 1 997) describes the pre-Edison elementary school as being in 
. .sad shape, plagued by some of the lowest grades and worst attendance.” The New York 
Times (Dec. 15, 1997), in the same vein, pictures Edison’s elementary school in Wichita as a 
“squat tan-and red-brick elementary school off a dirt road,” where “3 out of 4 students are living 
at the poverty level.” And an Edison news release (Dec. 15, 1997) touts big improvements in 
student achievement: 

Students have posted dramatic gains in reading and math on the Metropolitan 

Achievement Test (MAT/7). These first two-year gains come from Dodge-Edison 

Elementary School where students who have been at Dodge for two years have gained 

more than 25 percentiles against national norms in reading and math. 

However, the school that became Edison’s Wichita elementary school was not exactly 
what Time or the New York Times describes— and, if one looks at the test results in the way most 
researchers would, they are promising rather than “dramatic.” 

Notwithstanding the impression of shabbiness conjured up by the New York Times 
quotation, the school is a relatively new facility for Wichita, and it is one of the few schools in 
the district with air conditioning. Pre-Edison, it enrolled about 300 neighborhood students and 
housed a special education center that brought the enrollment up to about 400. Contrary to the 
Time article, the average daily attendance of 94.5 percent in the year before Edison took over 
matched the state average. 

Today, Edison’s Wichita elementary school enrolls about 600 students, and there is no 
special education center. Sixty-nine percent of the students qualify for free or reduced-price 
lunches. And while the student body is certainly poor, the impression of abject poverty created 
by Time and Newsweek is exaggerated. In fact, one in three Wichita schools has a higher 
percentage of free-lunch students than Edison, and, as mentioned earlier, the Edison school is in 
danger of losing the Title I federal funding it gets for serving disadvantaged students as a 
schoolwide project (http:/.%mv.aft.org,^research''edisonpioject/support\vicliita/1unch.htm) . 



As for the gain in test scores. Table A1 shows that the Edison school started off 3 NCEs 
behind the district in reading in grades 3 to 5, and after two years under Edison, it is 2 NCEs 
ahead of the district. In math, it started 2 NCEs behind the district average and after 2 years is 
now 7 NCEs ahead of the district. Compared to the district, these gains are small in reading and 
moderate in math. But to interpret them properly, we need to consider several other factors that 
Edison does not acknowledge. 
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Table A1 

District Assessment Data in Wichita 



Neighborhood Schools Neighborhood Magnets 







Dodge- 






Jeffer- 




Wash- 








Wichita 


Edison Caldwell 


Colvin 


son 


Payne 


ington 


Kellogg 


Ingalls 


% Minority 




34% 


69% 


56% 


37% 


34% 


70% 


31% 


64% 


% Free Lunch (96-97) 




76% 


84% 


92% 


85% 


81% 


75% 


63% 


69% 


% Free Lunch (97-98) 




69% 


81% 


90% 


76% 


82% 


91% 


68% 


na 


Teachers/100 students 




5.01 


7.15 


5.9 


5.63 


5.29 


6.10 


5.33 


5.77 


MAT7 Reading (NCEs) 




















Grade 3. Fall 1995 


50 


44 


53 


39 


41 


41 


59 


54 


45 


Grade 3. Fall 1996 


51 


48 


46 


48 


40 


48 


58 


48 


54 


Grade 3. Fall 1997 


52 


59 


51 


39 


47 


46 


43 


58 


na 


Change 


2 


15 


-2 


0 


6 


5 


•16 


4 


9 


Grade 4. Fall 1995 


51 


48 


47 


41 


47 


40 


55 


57 


48 


Grade 4. Fall 1996 


52 


49 


44 


39 


46 


50 


57 


61 


53 


Grade 4. Fall 1997 


53 


51 


51 


40 


48 


45 


51 


60 


na 


Change 


2 


3 


4 


•1 


1 


5 


-4 


3 


5 


Grade 5, Fall 1995 


53 


52 


51 


46 


41 


39 


59 


69 


52 


Grade 5. Fall 1996 


54 


55 


47 


44 


54 


48 


62 


54 


57 


Grade 5. Fall 1997 


54 


55 


47 


43 


51 


55 


59 


60 


na 


Change 


1 


3 


-4 


-3 


10 


16 


0 


•9 


5 


MAT7 Math (NCEs) 




















Grade 3. Fall 1995 


46 


44 


44 


43 


43 


41 


54 


61 


39 


Grade 3. Fall 1996 


50 


54 


43 


50 


41 


43 


60 


50 


57 


Grade 3. Fall 1997 


50 


61 


53 


37 


51 


50 


35 


59 


na 


Change 


4 


17 


9 


-6 


8 


9 


•19 


-2 


18 


Grade 4. Fall 1995 


47 


43 


50 


41 


44 


43 


55 


58 


48 


Grade 4. Fall 1996 


50 


51 


43 


43 


47 


53 


53 


55 


50 


Grade 4, Fall 1997 


51 


57 


51 


43 


48 


50 


50 


65 


na 


Change 


4 


14 


1 


2 


4 


7 


•5 


7 


2 


Grade 5, Fall 1995 


52 


50 


47 


43 


47 


37 


55 


66 


52 


Grade 5. Fall 1996 


53 


58 


45 


41 


51 


50 


60 


54 


59 


Grade 5, Fall 1996 


56 


60 


50 


43 


46 


60 


66 


56 


na 


Change 


4 


10 


3 


0 


-f 


23 


11 


•10 


7 


MAT7 Reading (NCEs) 




















Grades 3-5, Fall 1995 


51 


48 


50 


42 


43 


40 


58 


60 


48 


Grades 3-5, Fall 1996 


52 


51 


46 


44 


47 


49 


59 


54 


55 


Grades 3-5, Fall 1997 


53 


55 


50 


41 


49 


49 


51 


59 


na 


Change 


2 


7 


•1 


•1 


6 


9 


•7 


•1 


6 


MAT7 Math (NCEs) 




















Grades 3-5, Fall 1995 


48 


46 


47 


42 


45 


40 


55 


62 


46 


Grades 3-5, Fall 1996 


51 


54 


44 


45 


46 


49 


58 


53 


55 


Grades 3-5, Fall 1997 


52 


59 


51 


41 


48 


53 


50 


60 


na 


Change 


4 


14 


4 


•1 


4 


13 


-4 


•2 


9 



Source: School reports on WPS Web page, and information provided by school district officials. 
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1. Effects of new testing program. Some of the gain on the MAT/7 can probably be 
attributed to an effect that is well-known among researchers. After an initially poor showing on 
a new test, scores usually move upwards; and progress against national norms can occur merely 
because teachers and students become familiar with the tests. Even if one factors in the upward 
drift, Edison students did well. Nevertheless, their performance needs to be compared to district- 
wide scores and scores in control schools in order to separate out the gains that are unrelated to 
the Edison program. 

2. Comparison to similar neighborhood schools. Edison’s usual practice is to compare 
achievement in its schools to national norms only, but for a more precise evaluation, we should 
compare results in Edison schools with those in similar neighborhood schools. If that is done in 
Wichita, some of the advantage that Edison's Wichita school seems to have disappears. 

Caldwell, Colvin, Jefferson, and Payne are demographically similar neighborhood schools that 
receive Title I ftmding for disadvantaged pupils (see http://ww\v.usd259.com/elementary/elementai~y.html 
for school profiles).*^ All have the same or a larger proportion of minority children than 
Edison’s Wichita school, and all have a higher percentage of the student body getting fi-ee or 
reduced-price lunch. Pajme differs fi-om the other three control schools because, during the past 
two years (which also mark Edison’s involvement in Wichita), Payne, like Edison, has had a 
longer school year, and it has benefited fi-om extra resources for technology. Unlike Edison, 
however, it remains a neighborhood school. 

Results in the control schools are mixed. Over the two years between 1995 and 1997, 
Pajme improved a little bit more than Edison in reading— 9 NCEs compared to Edison's 7 NCEs— 
and a little bit less in math — 13 NCEs compared to Edison's 14 NCEs. Colvin, on the other 
hand, showed small losses in both math and reading. Caldwell and Jefferson improved in math 
as much as the district average (4 NCEs) but considerably below Edison, and Jefferson showed 
gains in reading of 6 NCEs, close to Edison's. 

3. Comparison to neighborhood magnet schools. Edison is really a neighborhood 
magnet school because a large number of its students come fi-om outside school boimdaries. 
Ingalls, Washington, and Kellogg are also neighborhood magnet schools, and in 1996-97, all 
three schools had lower percentages of students eligible for fi-ee or reduced-price limch than 
Edison. But while Edison’s percentage declined, theirs rose. At Washington, the percentage of 
students qualifying for fi-ee lunch increased fi-om 75 to 91 percent and at Kellogg fi-om 63 to 68 
percent; at Edison, the percentage declined fi-om 76 to 69 percent. Since, on average, 
achievement declines as the percentage of students eligible for fi-ee or reduced-price lunch 
increases, this information about student population needs to be factored in to any comparison 
between Edison and other neighborhood magnets. 



** The aborted Wichita State University evaluation 1996 identified eight anonymous control schools. The control 
schools used here— Caldwell, Colvin, Jefferson, and Payne— were the four schools identified by the school district 
for a subsequent evaluation, which also did not take place. 
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Figure A1 - Math Achievement on MAT/7 In Wichita for 
Dodge-Edison and Control Schools, Grades 3-5 




Colvin Jefferson Wash- Caldwell Wichita Payne ingaiis* Dodge- 
ington Edison 

* One-year trend. Schooi aboiished and repiaced with ingaiis-Edison 



Keiiogg 



Figure A2 -- Kansas State Grade 4 Math Assessment, Two-Year 
Trends at Dodge-Edison and Control Schools with Comparable 
Free/Reduced Price Lunch Students 




Dodge- Coivin Caidweii Keiiogg Jefferson ingaiis Payne Washington 
Edison 



Washington showed a decline in student achievement over this two-year period, and 
Kellogg stayed about the same. However, both schools have been performing at relatively high 
levels for several years. It’s likely, in Washington’s case, that the fall-off in achievement is 
associated with the steep rise in the percentage of students qualifying for free or reduced-price 
lunch over the same time period. Ingalls was abolished at the close of 1996-97 and reopened as 
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the second Edison elementary school in Wichita. Ironically, during its next-to-last year of 
operation, from fall 1 995 to fall 1 996, Ingalls improved math achievement as much as Edison, 
and it improved in reading about the same rate as Edison. The new Edison school replaced a 
program that was already effective and was getting better. 

4. Kansas State assessment program. There is another important source of 
information about achievement in Edison's first Wichita school, but the company has largely 
ignored it. In its first end-of-year report, Edison wrote that the Kansas State Assessments have 
provided useful findings and “baselines for year-two growth studies,” but the company is very 
selective about which data it reports. According to the company: 

First-year data indicated achievement typical of similar students. Only one of the state 
assessments, writing, was administered late enough in the school year (March) to 
provide evidence of first-year progress. Fifth graders at Dodge-Edison this year 
outperformed fifth graders at Dodge last year. But changes in the student population 
make it impossible to know whether these gains reflect additional learning or not. 

Edison’s Annual Report on School Performance (1997) indicates that Edison students also 
increased their writing scores during the second year of operation. However, Edison’s failure to 
talk about other test data from the state assessment program distorts the picture of student 
achievement. It’s true that the state reading tests for third graders and math tests for fourth 
graders, given in March and April 1996 and 1997, do not fully represent the impact of Edison 
over two full years. However, they provide a nearly complete two-year progress report, which 
shows no improvement in Edison’s first year and a little improvement in the second.'’ 

As indicated in Table A2, Edison shows a four-point improvement in expository reading 
compared to the pre-Edison year of 1995, and a two-point gain in math (see Figure A2). This 
showing is particularly feeble when you consider that the pre-Edison student body contained 
more minority students and had a slightly larger population qualifying for free lunch. Moreover, 
pre-Edison Dodge was a neighborhood school, not a neighborhood magnet school, and therefore 
did not benefit from the self-selection that characterizes the families whose children attend 
magnet schools. 



Although Edison includes the writing test of the Kansas State Assessments in its Annual Report on Student 
Achievement (while excluding and not even mentioning the reading and math tests), AFT has excluded the results 
from this report. When AFT asked the Wichita public schools for data on the Kansas State Assessments, no data 
were provided for the writing assessment. A school-district official indicated that the state writing assessment is 
given every other year and is scored by state-level personnel. However, Wichita also gives the assessment in 
alternate years. Alternate-year assessments are scored by school-level and district-level teams, and the scores are 
then averaged. The school district considers these scores to be for internal school use only and discourages the kind 
of use Edison made of them in its public report. Also, according to a district official with whom AFT spoke, the 
school-level scoring was much higher at the Edison school than the district scoring. Because the validity of the 
scoring is questionable and because we have no data on other Wichita schools, Edison’s scores on the Kansas 
writing assessments do not appear in our report. 
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Table A2 

Kansas State Student Achievement Data In Wichita 



Neighborhood Schools Neighborhood Magnets 



State 


Dodge 

94-95 


Dodge 

Edison Caldwell 


Colvin 


Jeffer- 

son 


Payne 


Wash- 

ington 


Kellogg 


Ingalls 


Enrollment 


415 


599 


403 


701 


279 


410 


369 


280 


515 


% Minority 


39% 


34% 


69% 


56% 


37% 


34% 


70% 


31% 


64% 


% Free Lunch (96-97) 


79% 


76% 


84% 


92% 


85% 


81% 


75% 


63% 


69% 


% Free Lunch (97-98) 




69% 


81% 


90% 


76% 


82% 


91% 


68% 


79% 


Average Daily Attendance 


94% 


na 


94% 


94% 


92% 


93% 


95% 


94% 




Teachers/100 students 5.82 5.01 

Third Grade Expository Reading (Percent Correct) 


7.15 


5.9 


5.63 


5.29 


6.1 


5.33 


5.77 


March, 1995 64 


50 




54 


51 


51 


46 


45 


62 


49 


March, 1996 64 




51 


59 


50 


52 


58 


58 


59 


51 


March, 1997 




54 


60 


70 


54 


53 


65 


64 


62 


T wo- Year Change 4 

Third Grade Narrative Reading (Percent Correct) 


6 


19 


3 


7 


20 


2 


13 


No test in 1995 




na 


na 


na 


na 


na 


na 


na 


na 


March, 1996 




46 


62 


48 


46 


55 


55 


59 


52 


March, 1997 




52 


60 


70 


55 


48 


61 


62 


58 


One-Year Char)ge 6 

Math-Fourth Grade Power (Percent Correct) 


-2 


22 


9 


-7 


6 


3 


6 


April, 1995 54 


44 




37 


42 


43 


40 


55 


51 


51 


April, 1996 56 




44 


58 


50 


50 


50 


53 


53 


48 


April, 1997 




46 


49 


47 


53 


57 


60 


49 


53 


Two-Year Change 




2 


12 


5 


10 


17 


5 


-2 


2 



Source: School reports on WPS web page and information provided by school district officials. 



The comparison between Edison and the four neighborhood schools and three neighborhood 

magnet schools on the Kansas State Assessments is far from impressive, as Table A2 shows. By 

March 1997: 

• Six of the seven schools had the same or better scores on expository reading, and Payne was 
only one point behind Edison. 

• Six of the seven schools had better scores on narrative reading. 

• All seven comparison schools scored better in math. 

• Over 2 years, five of the seven comparison schools improved their expository reading scores 
more than Edison. 

• In narrative reading, four of the seven comparison schools showed equivalent or greater 
improvement than Edison. 

• In math, only one of the seven schools improved at a rate lower than Edison. 
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The Wichita Eagle also called attention to the discrepancy between results for MAT/7 and 
the Kansas State Assessments (Dec. 26, 1997): . .numerous non-Edison Wichita elementary 

schools are doing as well or better than Dodge on the tests that count the most in Kansas: the 
assessment that the state uses to accredit schools.”^® 



“ One explanation for the relatively poor showing in math on the state assessments in comparison to the 
strong gains in math on the MAT/7 (see Table Al) could be Edison's use of frequent practice tests 
similar to the MAT/7. Wichita State evaluators wanted to examine test-taking practices at a set of 
control group schools to see if Edison’s “teaching to the test” was comparable to practices in other 
Wichita schools. However, the district refused to fund the continuation of the Wichita State evaluation, 
so the issue of teaching to the test at the Wichita Edison school cannot be resolved. 
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Exhibit A - Inconsistent Academic Gains 



AtDodge’-Edisofiy Big Gains on the MAT/7 for Third, Fourth and Fifth Graders . . . 

Over two years, reading scores improved by 7 NCEs, and math scores jumped by 14 NCEs. 
The 1997 scores surpass the national mean of 50, and are now comparable to Washington 
and Kellogg, two other neighborhood magnet schools with similar students. Wichita 
averages 53 in reading and 52 in math. 



Note: Fall-to-fall testing 
means that grade 3 scores 
reflect learning in grade 2, 
grade 4 reflects grade 3 
progress, and grade 5 reflects 
grade 4. SFA results for 
fourth graders shown below 
describe the grade 3 reading 
study from the prior year. 

The Kansas state math test for 
fifth graders refers to the prior 
year’s grade 4 test. 



Reading 




1995 1996 1997 Gain 


1995 1996 1997 Gain 



Grade 3 


44 


48 


59 


15 


44 


54 


62 


18 


Grade 4 


48 


49 


52 


4 


43 


51 


57 


14 


Grade 5 


52 


55 


56 


4 


50 


57 


62 


12 


Total -Edison 


48 


51 


55 


7 


46 


54 


60 


14 


Washington 


58 


59 


51 


-7 


55 


58 


50 


-5 


Kelloaa 


60 


54 


59 


-1 


62 


53 


60 


-2 
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« 3.5 

J 3.0 

> 2.5 

o- 2.0 
UJ 

a, 1.5 
2 1.0 
« 0.5 
0.0 




01995-96 

□1996-97 



Grade 2 to Grade 3 to 
Grade 3 Grade 4 



. . . But Third and Fourth 
Graders Fell Behind in the Primary 
Reading Studies. 

Despite above average scores on the 
MAT/7, the Success for All primary 
reading studies indicate that fourth graders 
performed below grade level and failed to 
gain one full year. Third graders also 
failed to keep pace and score at grade 
level. 



. . . And Math Scores of Fifth 
Graders on the Kansas State 
Assessment Lag Behind the State 
Average. The state average is 60 
percent correct compared to 46 
percent at Dodge-Edison. While 
showing superlative gains on the 
MAT/7, fifth graders have not 
improved on state assessments and 
score lower than students at such 
comparable neighborhood magnet 
schools as Kellogg and Washington. 




Dodge- Kellogg Wash- 
Edison ington 
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B. King Elementary School in Mt Clemens, Michigan 

In describing results at the Edison elementary school in Mt. Clemens, the company points 
proudly to results on the Iowa Test of Basic Skills (ITBS), a nationally normed, standardized 
test. The third graders in fall 1995 went from the 42nd percentile in math to the 55th percentile 
the following fall when they were in fourth grade (gaining about 7 NCEs). Reading improved 
from the 40th to the 51st percentile from grade 4 in 1995 to grade 5 in 1996 (for a gain of about 5 
NCEs). 



An Edison chart, reproduced as Exhibit Bl, tells this story graphically, and the story is a 
commendable one. It is therefore puzzling that Edison exaggerates its record by characterizing 
the first-year gains as “large” when they would only be “small to moderate” by Success for All 
standards (see table on p.27), and it also stretches the truth somewhat in saying that these gains 
are “corroborated by the state assessment, MEAP” (Annual Report on School Performance, 1997 
http://w\\w. edisoiiproiect.com/annualfranie.html) . 



Exhibit Bl 




1. Iowa Test of Basic Skills. But Edison’s presentation of Mt. Clemens's performance 
on the December 1996 ITBS is highly selective. It ignores poor ITBS results in third-grade math 
and reading— scores in the second year fell by 6 or 7 NCEs (see table Bl). It also fails to show 
that there was little improvement in fifth-grade math between December 1995 and December 
1996 (see figure Bl). Nor would you know that Edison students in third and fifth grades 
performed well below the national average in both years. In fact, Edison's chart is like a photo 
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that has been cropped to display the image in the most flattering way— accepted practice in 
photography but not in the presentation of data. Table B1 gives a fuller picture of progress in 
math and reading and also incorporates ITBS data measuring the second year of progress at Mt. 
Clemens. 

Table B1 

Iowa Test of Basic Skills, Mt. Clemens 



Reading Math Language 





1995 


1996 


1997 


1995 


1996 


1997 


1995 


1996 


1997 


Normal Curve Equivalents (NCEs) 














Grade 3 


44 


37 


47 


44 


38 


51 


40 


35 


40 


Grade 4 


44 


50 


48 


45 


52 


47 


38 


41 


40 


Grade 5 


43 


46 


50 


46 


45 


51 


36 


38 


44 


Change Since 1995 (NCEs) 
















Grade 3 




-7 


3 




-6 


7 




-5 


0 


Grade 4 




6 


4 




7 


2 




3 


2 


Grade 5 




3 


7 




-1 


5 




2 


8 


Number of Students Given ITBS 














Grade 3 


85 


79 


69 


82 


73 


67 


79 


72 


70 


Grade 4 


66 


87 


69 


63 


85 


65 


65 


85 


69 


Grade 5 


58 


85 


59 


49 


85 


57 


40 


84 


59 


Enrollment 




















Grade 3 




80 


79 




80 


79 




80 


79 


Grade 4 




87 


86 




87 


86 




87 


86 


Grade 5 




83 


84 




83 


84 




83 


84 


Percent of Enrollment Tested 
















Grade 3 




99% 


87% 




91% 


85% 




90% 


89% 


Grade 4 




100% 


80% 




98% 


76% 




98% 


80% 


Grade 5 




102% 


70% 




102% 


68% 




101% 


70% 



Source: Edison Project. Students are tested in December and f'iCEs are based on winter norms. 
* Figures add to more than 100% because the data are drawn from two different Edison sources. 



Over the two years from December 1995 to December 1997, Edison students improved 
by as little as 2 NCEs in fourth-grade math to as much as 7 NCEs in fifth-grade reading (Table 
Bl). Third and fifth graders improved in the second year while fourth graders fell back a little. 
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Figure B1 -Two-Year Change in Math and 
Reading (ITBS) In Mount Clemens 



60 
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Grade 3 Grade 4 Grade 5 Grade 3 Grade 4 Grade 5 
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01995-96 

11996-97 

11997-98 



There is another problematic feature of the ITBS tests at Mt. Clemens, which were 
administered by Edison (the school district has no standardized testing program). As Table B1 
shows, the number of students tested in grades 4 and 5 fell sharply between 1996 and 1997. In 
1996, all fifth graders were tested in math; the next year, only 68 percent were tested. In third 
grade, only 85 percent were tested in math, and in the fourth grade only 76 percent. 

If, as is the case in many school districts, the children excluded were special education 
students or students with limited proficiency in English, their exclusion should have tended to 
raise test scores. But whatever the reason they did not take the test in 1997, the sharp decline in 
the percentage of students tested in math in 1 997 (compared to 1 996) raises serious questions 
about testing protocol and testing results. 



2. Michigan Educational Assessment Program. Finally, Edison’s reporting of fourth- 
grade gains on the Michigan state assessments (MEAP) in reading and math between 1 995 and 
1 996 is odd, to say the least. On the one hand, Edison's statement comparing the two schools is 
clearly talking about the net differences between them: 

[CJhanges in MEAP scores of fourth graders from 1995 to 1996 exceeded those of other 
district^' fourth graders by 12 percentiles in math and 10 percentiles in reading, and of 
fourth graders statewide by .5 and 7 percentiles in math and reading respectively 
(Edison's Annual Report on School Performance, December, 1997 
bttp://www. edisonproiect.com/annualframe.htmn . 

And that's accurate enough. On the other hand, the chart Edison uses in its presentations to 
illustrate the point (Exhibit B2) misleadingly implies that Mt. Clemens made a 10-point gain in 

The Mt. Clemens district has only two elementary schools. 
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reading (the gain was 6 points but the control school lost 4 points) and a 12-point gain in math 
(Edison declined 2 points, but the control school declined 14 points). Presenting the data as 
Edison does in Exhibit B2 puts the best possible spin on mixed results. Not only does it make 
the reading gains look bigger; it also obscures the fact that Edison’s fourth graders lost ground in 
math. 



Exhibit B2 



Results: Mt. Clemens Ml 



Percentile Point Gain on MEAI* 
vs. Otltcr Elementary School 

Reading: +10.3 

Math: +12J 



1995: Behind in Both Subjects 
1996: in Botli Subjects 



This chart shows how Edison 
describes the MEAP results to the 
public. The reference to change in test 
scores is misleading because reading 
scores only improved 6 points and 
math declined 2 points. See text for 
details. 



Figiure B2 gives an accurate picture of the fourth-grade math results by graphing them. 
Also, by presenting data about Michigan state averages and about achievement in 1 994, before 
the school was managed by Edison, it adds two more points of comparison. 




-20 



Figure B2 -- Change in Grade 4 MEAP Math 
Using Graph to Show Changes 




Michigan Avg. 
^ King-Edison 
M Washington 



1994 Pre-Edison 1995 Pre-Edison 1996 Edison 
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Figure B3 -- Grade 4 MEAP Results For 
Mt. Clemens Elementary Schools 



1994 1995 Pre- 1996 1 994 Pre- 1 995 Pre- 1996 

Pre- Edison* Edison Edison Edison* Edison 

Edison 

* Edison began operation in fall 1995, but the MEAP, which is taken in the fall, 
describes achievement from the previous year. 

Figure B3 puts Edison's performance into a larger district and state perspective. It shows 
that while the percentage of students doing math at the satisfactory level declined during 
Edison’s first year (1996), it had increased in previous years by substantial margins (see also 
Table B2). On the other hand, reading scores declined precipitously from 60 percent satisfactory 
in fall 1994 to less than 40 percent satisfactory in fall 1995 (see figure B3). Edison restored 6 
percentage points in its first year of operation, but reading performance was still lower than the 
year before Edison. Michigan has changed the MEAP from a fall test to a January test, and new 
results will be available in summer 1998. 

The grade 5 science and reading tests have now been given twice, allowing an assessment 
of how well Edison is doing compared to other schools in the district and the state average, as 
well as how much achievement has improved during the Edison program's second year. Spring 

1996 is the first time that the fifth-grade tests have been given in Michigan, so there is no good 
baseline data on which to assess the effect of Edison's first year. However, these tests show 
somewhat better performance by the control school. Both schools performed well below the 
state average in science in 1996. The control school nearly caught up with the state average in 

1997 while the Edison school showed no improvement. Only 20 percent of Edison students 
performed satisfactorily in 1997. On the writing test, the control school performed better than 
Edison in 1996, but both schools had about 69 percent of their students performing at the 
satisfactory level in 1997— almost equal to the state average. 



Michigan 
■Washington 
P King-Edison 
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Table B2 

State Assessments (MEAP) In Michigan and Mount Clemens 
Grade 4 





1993 


Pre-Edison 

1994 


1995* 


Edison 

1996 


Enrollment 










Edison 




484 


490 


508 


Washington 




511 


529 


504 


Percent Free Lunch 










Edison 




59% 


48% 


48% 


Washington 




54% 


52% 


55% 


Expenditures Per Pupil 










Edison 




$4,499 


$5,789 




Washinaton 




$5,173 


$5,662 




Grade 4 Math (Fall)-Percent Satisfactory 






Michigan Average 


48.9 


61.6 


63.1 


60.5 


Edison 


37.1 


49.1 


57.4 


55.3 


Washington 


32.9 


71.4 


62.8 


48.4 


Grade 4 Reading (Fall)-Percent Satisfactory 






Michigan Average 


43.6 


43.6 


49.9 


49.0 


Edison 


35.9 


61.8 


39.7 


46.1 


Washington 


25.6 


46.4 


43.0 


39.1 


Grade 5 Science (Spring)-Percent Satisfactory 






Michigan Average 






36.9 


36.8 


Edison 






20.4 


20.3 


Washington 






24.2 


35.6 


Grade 5 Writing (Spring)-Percent Satisfactory 






Michigan Average 






55.6 


73.4 


Edison 






40.4 


68.9 


Washington 






51.6 


69.0 



Source: Michigan Department of Education, http://www.mde.state.mi.us/reports/meap/ 
* Edison began operation in fall 1995, but the MEAP, which is taken in the fall, 
describes achievement from the previous year. 



In analyzing the results, we should also keep in mind that Edison had an advantage that 
is not related to its academic program. As already noted above, the percentage of students 
qualifying for free lunch dropped from 59 percent before Edison to 48 percent under Edison, 
demonstrating that Edison had a somewhat less disadvantaged population than the pre-Edison 
King Academy. 
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C. Washington Elementary in Sherman, Texas 



The Edison Project has been remarkably quiet about its school in Sherman, Texas, which 
opened in 1995. At the end of the first year, the company’s Web site said only that data 
measuring Sherman’s academic growth would be available the following June. In its 1997 
Annual Report on School Performance ( http://wvvw.edisonproject.conT/aimualfTame.htoil) , Edison said 
that, while student achievement dropped in the third grade, there were improvements in both 
fourth and fifth grades. Viewed over the entire two-year period between September 1995 and 
June 1997, however, the results at Sherman are considerably less promising than these statements 
indicate. 

During Edison’s first year, reading scores on the Texas state assessment (TAAS) declined 
sharply in third grade, and so did fourth-grade math scores. Edison subsequently improved math 
scores in grade 4, but second-year scores were still below what they were before Edison took 
over. During the second year, grade 3 reading scores stayed about the same, and math declined 
further. Fourth-grade reading rose in both years. However, scores in all grades were well below 
the state average, as well as the group of demographically comparable schools fi-om across Texas 
established as control schools by the Texas Education Agency (the “Texas control group”).^^ 

What is the reason for the decline? It is not demographics. Although the Edison school 
is Sherman’s poorest and most ethnically diverse school, its demographics did not change much 
after Edison took over. As shown in Table Cl, the percentage of Afiican- American and Hispanic 
students declined, and the percentage of students classified as economically disadvantaged also 
decreased slightly. 

The percentage of students excluded fi"om TAAS for special education reasons was 
approximately the same in math and reading in Edison’s first year as in the year just before 
Edison took over. However, the percentage excluded was considerably higher in the second year 
(up fi-om 2.2 percent to 8.4 percent in reading; up fi-om 1.8 percent to 8.4 percent in math). The 
fact that this did not lead to higher test scores suggests that Edison students performed even more 
poorly than the numbers indicate. Edison excluded fewer students in writing than had been the 
case pre-Edison, which may explain some of the decrease in the grade 4 writing test. 

Table C2 shows the percentage of Sherman students passing the TAAS reading, writing 
and math assessments fi-om 1994-95, the year before Edison took over, through 1996-97, the last 
year for which data are currently available. And it compares the Edison figures with the school 
district and state percentages, as well as the percentage of students passing in the Texas control 
group. 



In the spring of Edison’s first year, the percentage of students who passed grade 3 
reading fell fi-om 88 percent to 67 percent and in math fi-om 68 percent to 66 percent. Students in 



The state of Texas matches each school with 100 other demographically comparable schools in the state. One 
would expect an Edison school to perform at the control-group average in order for the Edison program to show no 
negative effect. 
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the control-group schools dropped 4 points in reading (compared with Edison's 22 points). And 
in the math assessment, the percent of control-group students who passed rose from 68 to 75. 

In fourth grade at the Edison school, the reading pass rate improved from 50 to 56 
percent, but math scores slid from 52 percent passing to only 34 percent. However, the control 
group of similar schools was less impressive in relation to Edison, at least in reading. Their 
reading scores declined (from 75 percent passing to 69 percent), though they still exceeded 
Sherman's 58 percent. The percentage of control group students passing math rose from 68 to 71 
percent (compared to Edison’s 18-point decline). 



Table C1 

Demographics of Edison School In Sherman, Texas 

Pre-Edison Edison Edison 





1994-95 


1995-96 


1996-97 


Black 


20.5% 


16.8% 


18.3% 


Hispanic 


22.4% 


21.8% 


21.8% 


White 


57.7% 


58.8% 


59.2% 


Economically 

Disadvantaaed 


70.1% 


63.9% 


65.5% 


TAAS Exempted for Special Education 
Reading 2.2% 


2.2% 


8.4% 


Writing 


8.5% 


1.7% 


4.3% 


Math 


1.8% 


1.5% 


8.4% 


TAAS Exempted for Limited-English Proficiency 




Reading 


na 


0.0% 


5.0% 


Writing 


na 


0.0% 


0.0% 


Math 


na 


0.0% 


5.0% 



Source: Campus Reports of Academic Excellence Indicator 
System, Division of Performance Reporting, Texas Education 
Agency. 

(http://www.tea. state. tx.us/’perfreport'aeis/97/canipus. srch.html) 
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Table C2 

TAAS-Percent Passing 



Pre- 





Edison 


Edison 


One-Year 


Edison 


Two-Year 




1994-95 


1995-96 


Change 


1996-97 


Change 


Third Grade TAAS 












District (Sherman) 












Reading 


82.8 


80.3 


-2.5 


81.5 


-1.3 


Math 


74.3 


70.1 


-4.2 


76.9 


2.6 


Texas Control Group 












Reading 


78.8 


75.0 


-3.8 


81.9 


3.1 


Math 


68.5 


75.3 


6.8 


85.0 


16.5 


State Average 












Reading 


79.5 


80.5 


1.0 


81.0 


1.5 


Math 


73.3 


76.7 


3.4 


81.0 


7.7 


Edison 












Reading 


88.6 


68.0 


-21.9 


69.5 


-19.0 


Math 


68.2 


66.2 


-2.0 


53.8 


-14.4 


Fourth Grade TAAS 












District (Sherman) 












Reading 


78.6 


77.5 


-1.1 


83.0 


4.4 


Writing 


86.1 


76.0 


-10.1 


81.8 


-4.3 


Math 


68.1 


66.8 


-1.3 


74.3 


6.2 


Texas Control Group 












Reading 


75.6 


69.4 


-6.2 


81.7 


6.1 


Writing 


83.9 


80.5 


-3.4 


86.7 


2.8 


Math 


68.1 


71.7 


3.6 


84.0 


15.9 


State Average 












Reading 


79.0 


78.3 


-0.7 


82.5 


3.5 


Writing 


84.0 


86.5 


2.5 


87.1 


3.1 


Math 


70.0 


78.5 


8.5 


82.6 


12.6 


Edison 












Reading 


50.0 


56.0 


8.0 


64.5 


14.5 


Writing 


69.4 


66.7 


-0.7 


53.2 


-16.2 


Math 


52.0 


34.0 


-18.0 


48.4 


-3.6 



Source: Campus Reports of Academic Excellence Indicator System , Division of 
Performance Reporting, Texas Education Agency. See Table C1 for Web citations 
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Figure C1 -- Percent of Third Grade Students 
Passing TAAS In Sherman Texas 



O) 

c 

'«) 

(A 

CO 

D. 



C 

0 > 

■o 

3 

Si 



c 

0> 

u 

L_ 

0> 

Q. 



100 

90 

80 

70 

60 

50 

40 

30 

20 

10 

0 



1995 1996 1997 

Pre- Edison Edison 

Edison 



Instate 

B Control Group 
Q Edison 



1995 1996 1997 

Pre- Edison Edison 

Edison 




Figure C2 -- Percent of Fourth Grade Students 
Passing TAAS In Sherman Texas 




^ state 
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Q Edison 
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In Edison’s second year, grade 3 reading improved just 2 points, and math dropped from 
66 to 54 percent passing. In grade 4, reading improved again, and over two years, the percent 
passing increased from 58 percent to 65 percent. However, it is still 15 points below the control 
group average of 82 percent. 



In math, Edison fourth graders also rebounded, but with a 48 percent satisfactory rating, it 
lags far behind the control group average of 84 percent. If students are followed diagonally from 
third grade in 1996 to fourth grade in 1997, the way Edison prefers to track test scores in most of 
its other schools, the percent of students passing reading decreased a little, from 68 to 64.5, while 
in math, the satisfactory rating slumped from 66 to 48 percent. The control group gained 7 



ERIC 



73 



64 





points in reading and 12 points in math when 1997 grade four scores are compared with 1996 
grade three scores (see table C2). 

There are a number of possible explanations for Sherman's disappointing performance. 
Edison has said that school leadership is the problem. That could well be the case— it is as big a 
problem in many of this country’s troubled schools as poor curriculum. And it is something that 
Edison will need to be able to deal with if it is going to demonstrate how to turn around low- 
performing schools. 

Some details about staffing may also be significant. Although the number of students 
enrolled went up between Edison's first and second years, the number of teachers decreased, 
fi-om 33 to 30; the number of aides decreased fi-om 8 to 2; and the percentage of teachers with 
less than 5 years experience increased fi'om 63 percent to 69 percent (compared to 30 percent 
elsewhere in the district [TEA, 1997]). 



Exhibit C 




Source; Edison Project handout 



The state of Texas placed the Edison school on its list of low-performing schools for part 
of 1997 based on low performance of Hispanic students in math. Of nearly 6,000 schools in 
Texas, only about 70 are classified as low performing. (See 

http://www.tea.state.tx.us/accountability/rat97/ for details on the Texas Accountability Rating System. 
See Appendix (page 91) for more details about Sherman's status.) 
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D. Reeves Elementary School in Dade County 

The contract between Edison and the Dade County school system calls for an evaluation 
conducted by the school system’s Office of Educational Evaluation. At this writing, it is the only 
comprehensive evaluation of student achievement planned for any of the 1 3 schools Edison 
opened in its first two years of operation. Moreover, the research design is based on the kind of 
solid evaluation standards generally lacking with the Edison program. 

The Stanford Achievement Test (SAT) and the Florida Writing Assessment are used as 
outcome measures, and the 1995-96 administration of the SAT, the spring before Edison began 
operation, will function as the baseline. Since students are not tested until they are in the first 
grade, the evaluation will deal only with grades two to five. For the same reason, the analysis 
will be limited to children who enrolled the first year of the program and were able to take the 
pretest the preceding spring. 

Student achievement in the Edison school will be compared to achievement in “virtual 
schools” of comparable students. This control group, which has been selected in a way similar to 
the control groups for Edison’s Primary Reading Studies in Wichita and Sherman, is a stratified 
random sample corresponding proportionally to the following student characteristics: grade 
level, ethnicity, eligibility for free or reduced-price lunch, and performance on the pretest. 

Results of the first-year evaluation, which has not yet been released, are characterized, 
both by Edison (in its year-end report for 1996-97) and the author of the evaluation, as showing 
no difference between Edison and the control group in reading. In math, statistically significant 
differences favored the control group. The release of this evaluation, which was postponed from 
January 1998, will not take place until the spring of 1998, and no other details regarding student 
achievement are currently available. 
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E. Roosevelt Elementary in Colorado Springs 



The Edison charter school in Colorado Springs was established in 1996 in close 
cooperation with the Colorado Springs school district. While enrolling some out-of-boimdary 
students, Roosevelt is primarily a neighborhood school. All students in the Colorado school 
district, including those in the Edison school, are assessed using a series of tests, the District 
Achievement Level Tests (DALT), that have been created for the district. Because they are local 
tests, they do not have any national norms. 

Table El compares results for Edison’s first year with the results in Roosevelt for the 
preceding four years and the average of those four years. The first-year scores are lackluster 
when compared with scores from the year before Edison took over— these scores that would 
ordinarily be considered a baseline — and even less impressive when compared with the four-year 
average. 



Table E1 

Trends In Student Achievement At Roosevelt Elementary 





Gain From Fall to Sprinq 


Average 


Edison 




92-93 


93-94 


94-95 


95-96 


92-96 


96-97 


SubjVGrade 

Reading 3rd 


6.6 


7.1 


10.7 


5.0 


7.4 


5.4 


Reading 4th 


3.7 


9.1 


6.5 


4.9 


6.1 


2.5 


Reading 5th 


1.3 


4.7 


4.6 


3.6 


3.6 


3.4 


Language 3rd 


5.9 


8.8 


9.5 


10.1 


8.6 


5.7 


Language 4th 


5.4 


9.6 


8.1 


5.0 


7.0 


6.4 


Language 5th 


7.0 


3.1 


2.8 


4.0 


4.2 


5.2 


Math 3rd 


11.3 


9.8 


9.3 


8.9 


9.8 


2.8 


Math 4th 


8.1 


8.7 


7.0 


8.6 


8.1 


4.8 


Math 5th 


6.1 


8.3 


4.8 


3.4 


5.7 


4.1 



Source: District data compiled by Colorado Springs Education Association. 

At this point, however, the real story is Edison’s presentation of the data so that Edison 
students appear to surpass pre-Edison students and even the district average. Since, as Table E2 
shows, the pre-Edison school lagged behind the district in reading, language, and math in grades 
three, four, and five, moving ahead of the district average would be an impressive 
accomplishment— if it were true. Exhibit E shows what Edison is claiming in the way of 
improvement for its fourth graders. How can the figures look so different from those in Figure 
El, which shows results for fourth grade reading and math? 
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Figure E1 - Change In Student Achievement From 
Fall to Spring In Colorado Springs 
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Exhibit E 



DALT Results 1995/6 to 1996/7 
Roosevelt-Edisou and District Eleven 
Current Fourth Graders 




Di5lru'1 KoiiMfvclt-EUiMin DiMriti Ittnjjevtfh-ndi.Mtn Disirict Knoscvi'lt-Filisuri 

Reading Language Mathematics 



Source: Edison Project handout 



Schools in Colorado Springs test in the fall and then again in the spring, and their results 
are presented in terms of gains between the two tests. Thus, first-year results at Roosevelt would 
be the difference between scores in fall ‘95 and spring ‘96 (see Table E2). As already mentioned. 
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Edison argues that these results do not capture the effect of their longer school year, and that is a 
legitimate complaint. Edison has dealt with the problem by negotiating both earlier fall and later 
spring test dates for 1997-98. For 1996-97, however, Edison has decided that the best way to 
present the data is from spring '96 to spring '97. And it is this decision that leads to most of the 
difference between the results shown in Figure El and Exhibit E. 

Table E2 

Different Ways to Compare Testing Data at Roosevelt 

Diagonal Movement (b) 













Gain (c) 


Gain (c) 


Gain (d) 


Gain (e) 




Pre-Edison 


Edison 


Fall '95 to 


Fall '96 


Spg. '96 


Spg. '96 




Fall '95 


Spg. '96 


Fall '96 


Spg. '97 


Spg. '96 


Spg. '97 


Spg. '97 


Fall '96 


READING 


















Edison 


















Third 


182.4 


187.4 


184.2 


190.1 


5.0 


6.0 


a 


a 


Fourth 


191.8 


196.7 


195.5 


198.2 


4.9 


2.7 


10.8 


8.1 


Fifth 


198.7 


202.3 


201.4 


204.8 


3.6 


3.4 


8.1 


4.7 


District 


















Third 


191.4 


198.8 


191.1 


198.3 


7.4 


7.2 


a 


a 


Fourth 


200.0 


205.5 


199.1 


205.1 


5.5 


6.0 


6.3 


0.3 


Fifth 


206.5 


209.9 


206.1 


210.3 


3.4 


4.2 


4.8 


0.6 


LANGUAGE 


















Edison 


















Third 


180.2 


190.3 


184.4 


190.7 


10.1 


6.3 


a 


a 


Fourth 


195.2 


200.2 


194.2 


200.6 


5.0 


6.4 


10.3 


3.9 


Fifth 


196.7 


200.7 


202.0 


207.1 


4.0 


5.1 


6.9 


1.8 


District 


















Third 


190.9 


200.8 


190.5 


200.3 


9.9 


9.8 


a 


a 


Fourth 


201.3 


208.1 


200.9 


207.8 


6.8 


6.9 


7.0 


0.1 


Fifth 


207.9 


212.5 


208.3 


213.4 


4.6 


5.1 


5.3 


0.2 



MATH 

Edison 



Third 


175.4 


184.3 


177.7 


181.0 


8.9 


3.3 


a 


a 


Fourth 


190.0 


198.6 


191.6 


196.5 


8.6 


4.9 


12.2 


7.3 


Fifth 

District 


199.2 


202.6 


200.6 


204.7 


3.4 


4.1 


6.1 


2.0 


Third 


185.9 


198.3 


185.1 


196.9 


12.4 


11.8 


a 


a 


Fourth 


196.6 


208.0 


197.5 


207.5 


11.4 


10.0 


9.2 


-0.8 


Fifth 


205.2 


215.3 


206.4 


215.8 


10.1 


9.4 


7.8 


-1.6 



a. Diagonal movement can only be calculated for students moving from grade 3 to grade 4 and grade 4 to grade 5. 

b. Column entries for fall of grade 4 or grade 5 represent gains from spring of grade 3 or grade 4. 

c. How school district measures student progress. 

d. How Edison would like to compare student progress. 

e. At Roosevelt, almost all of the changes from spring to fall reflect changes in the type of students. 
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While student achievement, over seven months from fall ‘96 to spring ‘97, increased by a 
score of 3 to 6 points in reading and 3 to 5 points in math (depending on grade level). May, 
June, and the summer vacation netted an additional 2 or 3 points in reading and 5 to 7 points in 
math (see Table E2). These are astonishing gains, even given Edison’s longer school year— 
especially considering that spring to fall results are generally flat, as evident in the district 
averages in Table 2. The more likely explanation of the spring to fall jump is the change in 
student population: The percentage of students in the Edison school eligible for free or reduced- 
price lunch dropped from 54 in the spring before Edison took over to 44 in the fall of 1 997, 
according to school district officials. In other words, a significant number of the students who 
entered and were tested in the fall were more advantaged. The Edison program apparently had 
little to do with the big jump in test scores-except insofar as it attracts higher scoring students.^^ 

We can’t know whether Edison realized that this way of presenting data is questionable, to 
say the least. But it seems like another example of the tendency to put the best face on things— 
even if that involves squeezing the truth— that characterizes much of the data reporting in 
Edison’s early schools. 



“ When the fall 1997 DALT scores became available, Edison switched from the spring-to-spring comparison to a 
fall-to-fall comparison. The new comparison tells about the same story as the one described in Exhibit E (not much 
difference from district gains), and it also suffers from the same problem. Achievement gains over the summer are 
about as large as achievement gains during the school year. The AFT will incorporate the fall 1997 scores into this 
analysis when the spring 1998 data become available. 
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F. Boston Renaissance Charter School 



Edison has made some pretty big claims about its school in Boston. For example, the company 
asserts that early tests showed two years’ growth in just one year. On the Massachusetts 
Educational Assessment Program (MEAP) assessments in 1996, according to Edison, its charter 
school students exceeded the norms for comparable students by a wide margin in reading and 
science and by smaller margins in mathematics and social studies. However, the MEAP results 
are far less clear cut— and less favorable— than Edison indicates, and after two years, the whole 
picture of achievement at Edison’s Boston charter is extremely confused. One reason is that 
Edison has used a number of different tests to assess student achievement, shifting from one to 
another, and on at least one occasion it used inappropriate tests and test norms. There is also a 
question about the percentage of students actually taking many of these tests. As a result, it is 
hard to get a coherent picture of student achievement at the school— and it is hard to see how 
Edison does. 

1. 1996 state assessments. In April 1996, all Massachusetts fourth graders took a 
comprehensive battery of achievement tests covering reading, mathematics, science, and social 
studies known as the Massachusetts Educational Assessment Program (MEAP). Though Edison 
says that the students at its Boston charter school achieved outstanding results in these tests, 
several factors cast doubt on this claim. Comparing charter school and public school results in 
Massachusetts is very problematic because charter schools have more leeway in selecting 
students and in excluding them.^'* Perhaps more important, schools of choice attract especially 
motivated parents, and the effect of motivated parents on their children's achievement can be 
mistakenly attributed to the school. In any case, Edison seems to have exercised selectivity in 
the students it tested, as shown in Table FI. 

Indeed, Edison’s good results on the 1996 MEAP may have as much to do with which 
students did not take the test as with Edison’s preparation of the students who did. Only 8 1 
percent of Edison fourth graders took the MEAP, compared to the state average of 90 percent. 
One reason for the low participation rate is that only 27 percent of Edison special education 
students were tested, compared with 52 percent statewide. Since Edison apparently tries to limit 
special education children to those who can be taught mainly in regular classroom settings, its 
high exclusion rate is especially surprising. In addition, Edison test-takers did not include any 
LEP students. Of the remaining students— in other words the great majority— Edison tested only 
90 percent, compared to the 97 percent state average. 



The effect of this selectivity is evident in some facts about demographics at Edison’s Boston and Worcester 
schools. Between its first and second years, the percentage of students in Edison’s Boston charter school eligible for 
free or reduced-price limch fell from 64 percent to 50 percent. In 1995-96, one-third of the school enrollment came 
from private schools. In Edison’s Worcester school, nearly one-fifth of its students, who had originally been chosen 
by lottery, returned to Worcester public schools by mid-year. 
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Table F1 

Number of Students Tested, 1996 Grade 4 MEAP 





Number 

Enrolled 


Students 

Tested 


Percentage 

Tested 


State 








All Students 


71 ,023 


63,732 


90% 


Special Needs 


9,749 


5,068 


52% 


Limited English 


2,666 


1,655 


62% 


Other 


58.608 


57.011 


97% 


Rennaisance Charter School 






Ail Students 


108 


88 


81% 


Special Needs 


15 


4 


27% 


Limited English 


0 


0 


0 


Other 


93 


84 


90% 



Source: Massachusetts Department of Education 



Testing fewer students, especially when they are special needs or LEP students, is likely 
to translate into an achievement advantage. The company also misstated the results when it 
claimed that scores at the Boston charter “exceeded the norms for comparable students by a 
wide margin in reading and science.” According to the state of Massachusetts, test differences of 
less than 50 points are not significant. And the differences between Edison averages and the 
midpoint of the statistically constructed control group of comparable students were right around 
this 50-point difference except in reading, where the difference was 70 points (see Table F2). At 
best, the difference would be considered small. 

Table F2 

Edison's 1996 MEAP Compared to State and Comparison Band 

Social 





Reading 


Math 


Science 


Studies 


Boston Edison 


1350 


1320 


1340 


1330 


State Average 


1350 


1330 


1360 


1340 


Comparison Band* 


1240-1310 


1240-1300 


1280-1320 


1250-1310 


Midpoint 


1280 


1270 


1300 


1280 



‘Expected average score for BRCS based on regression with three variables: 
the percentage of students receiving free or reduced-priced lunch, family 
language background, and preschool experience. 



The comparison score band was developed by the state (based on a statistical procedure 
called regression analysis) using three variables predictive of student achievement: (1) the 
percentage of students receiving fi^ee or reduced-priced lunch (self-reported by the principal), (2) 
family language background, and (3) the percentage of students who had attended preschool 
(self-reported by the students). At the grade 4 level, parental education is not part of the model. 
There are no statistical controls for the motivation and school involvement of parents. 
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An Edison bar graph, used in promoting the company’s program, also exaggerates the 
Edison school’s MEAP achievement by a creative use of scale (compare Figure FI and Exhibit 
FI, which is the Edison promotional piece). 



Figure F1 1996 MEAP Results for Boston-Edison, 
State Average and Control Group 
(Graph with Scale Beginning at Minimum Score) 



1600 
1500 

O 

h- 

<§ 1400 

*S 

g 1300 

CO 

< 1200 
E 

1100 
1000 

Reading Math Science Social 

‘Midpoint of comparison band (See Table F3) 




Q Boston Edison 
■state Average 
P Control Group* 



Note: According to the state of 
Massachusetts, differences of 
less than 50 points are not 
"educationally significant." 



Exhibit FI 



Results: Bo.stou MA 



l^crformrtncc vcraiis Ejtpcctntion on MEAP 







This overhead, distributed by 
Edison, displays the same data as 
Figure FI except by restricting the 
scale from 1,220 to 1,360, instead 
of from the scale score minimum 
of 1,000 to the scale score 
maximum of 1,600, the 
educationally marginal 50 point 
differences appear striking. In 
addition, the reading and social 
studies scores for Edison in this 
chart should be reversed with 
reading equal to 1,350 and social 
studies equal to 1,330. 




82 



73 



2. Standardized testing. A confusing array of tests and testing dates at Edison’s Boston 
charter school also makes it difficult to get a clear picture of student achievement (see Table F3). 
Edison uses two different standardized tests. The MAT/7 is given in the fall and provides data 
assessing Edison's first two years. The Stanford 9 is given in the spring. Some of the 
standardized tests indicate that students are making great progress. However, the exaggeration 
already evident in Edison’s presentation of the 1996 MEAP results is also in evidence here. 



Table F3 

Standardized Testing at Boston Charter School 



Reading Math 







Grade 3 


Grade 4 


Grade 5 


Grade 6 


Grade 3 


Grade 4 


Grade 5 


Grade 6 


A. MAT/7 




















Grade Equivalent 


















Oct-95 


MAT/7 


2.7 


3.2 


4.8 




2.0 


2.3 


4.2 




Oct-96 


MAT/7 (a) 


no data 


4.5 


5.0 


7.6 


no data 


4.2 


4.1 


7.4 


Oct-97 


MAT/7 


3.4 


4.3 


5.6 


7.0 


2.9 


3.8 


4.6 


5.6 


NCEs 




















Oct-95 


MAT/7 


45 


42 


52 




37 


29 


42 




Oct-96 


MAT/7 (a) 


no data 


57 


54 


63 


no data 


53 


40 


64 


Oct-97 


MAT/7 


55 


52 


54 


53 


50 


47 


48 


42 


B. Stanford/9 


















Grade Equivalent 


















Apr-96 


Stanford 


3.9 


no data 


6.9 




4.0 


no data 


5.7 




Apr-97 


Stanford (b) 


no data 


4.4 


5.2 


6.5 


no data 


4.3 


5.2 


6.0 


NCEs 




















Apr-96 


Stanford 


52 


no data 


56 




54 


no data 


50 




C. Massachusetts Reading Assessment 














NCEs 




















Apr-97 


ITBS (c) 


47 


na 


na 




na 


na 


na 





Source: Detailed results from the test publishers provided by the Edison Project. 

a. Grades 4 and 5 and 6 were given the grade 3, 4 and 5 test forms and scored using spring norms. 

b. Derived from charts in the annual report to the Massachusetts charter school agency. 



No data on number of students tested, and these data are not verified by test publisher data, 
c. NCE based on percentile ranking of 45. 

Edison made two claims of early success, the first legitimate and the second questionable. 
First, the spring 1996 Stanford/9 (panel B, Table F3) was compared to the MAT/7 given the 
previous fall when the school first opened (Panel A), and this indicated that students had 
progressed one-and-one half to two grade levels in one year. 

Second, Edison claimed that its students made big gains on the MAT/7 between April 
1996 and October 1996. For example, fifth-graders moving to sixth grade went from 6.9 grade 
equivalents in reading on Stanford the prior spring to 7.6 grade equivalents on the MAT/7 the 
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next fall (second row of Table F3); and from 5.7 grade equivalents to 7.4 grade equivalents in 
math. 



On the second point, the gain of almost two full years in only three months of school 
(May, June, and September), broken by summer vacation, is as surprising and implausible as the 
summertime gain in Colorado Springs (see page 69). However, the answer here seems to lie with 
testing irregularities rather than changes in the student population. In fall 1996, fourth-grade 
students were tested with the third-grade test using spring grade 3 norms, fifth graders were 
tested with the fourth-grade test using spring grade 4 norms, and sixth graders were tested with 
the fifth-grade test using spring grade 5 norms. In the Edison Project’s (1997) Aimual Report on 
School Performance, Edison argues, unconvincingly, that the grade equivalent data are 
legitimate, but not the NCE or percentile rankings. 

There are also problems with the two-year success rate that Edison claims. Over two 
years, the MAT/7 tests show scores improving by about 10 NCEs in reading in grades 3, 4, and 
5 and by larger amounts in math in grades 3 and 4. Most of the changes, however, seem to have 
occurred in the first year of the Edison program. Based on grade equivalents, as Edison suggests, 
and using the fall 1996 data, fourth and sixth grade showed declines in reading and in math in 
year 2. Grade five improved by half a grade equivalent in both math and reading. 

Edison's progress in the second year of operation can also be measured by comparing the 
spring 1997 Stanford/9 to spring 1996 scores. These data indicate an achievement decline during 
Edison's second year. Edison has not been forthcoming with the spring 1997 Stanford testing 
data. Results were not published in their Aimual Report on School Achievement, and the 
detailed data from the test publisher have not yet been provided to AFT. The 1997 information 
comes from charts submitted to the Massachusetts state chartering agency and provides 
information only in grade equivalents. In one year, students gained one-half a year or less as 
they moved from third grade in 1996 to fourth grade in 1997 and from fifth grade to sixth grade. 
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Exhibit F2 



The fall 1996 administration 
of the MAT/7 makes Edison 
students look terrific, 
especially the sixth graders, 
who appeared to gain three 
grade equivalents in one year. 
However, testing irregularities 
in fall 1996 invalidate the 
results. Fourth-grade students 
were tested with the third- 
grade test using spring grade 3 
norms, fifth graders were 
tested with the fourth-grade 
test using spring grade 4 
norms, and sixth graders were 
tested with the fifth-grade test 
using spring grade 5 norms. 



nriston CJiartcr Schiml and M^slh MAT-?* Scores 

Ovc/ Time 
F5U 1995/FaU 199^ 

Crede 




kljlii Mjtli 

ThiAJ fen Crjde F'lfeurK to E-lflh Tifth tn SUth 

Tm: 7 




Source: Edison Project handout 



Moreover, the issue of skewed results caused by excluding a significant number of 
students from testing, which came up with the 1996 MEAP, has continued to be a problem, as 
indicated by Table F4. Results for only about 75 to 85 percent of students are reported for any of 
these tests. This is a very low percentage, and it cannot be explained away by talking about 
special education students who are severely disabled. The student body at the Boston charter 
includes only 1 1 percent of special education students, and nearly all of them spend most of their 
time in regular classrooms. The apparent pattern of excluding a large percentage of students 
from testing cannot be generalized to all other Edison schools. In Mt. Clemens, where Edison 
runs its own standardized testing program, there were few exclusions in 1995 and 1996, though a 
high percentage were not tested in 1997. In Wichita in 1995, almost all students were tested. 
However, test exclusion is a pattern in Boston, and one that is likely to improve Edison’s overall 
scores without necessarily indicating improved student achievement. 




Table F4 

Students Tested at Boston Charter School 



Students Percent Tested 

Grade 3 Grade 4 Grade 5 Grade 6 Grade 3 Grade 4 Grade 5 Grade 6 

Enrollment 



1995-96 (estimated) 


112 


112 


1996-97 (Apr 97) 


114 


112 


1997-98 (Jan 98) 


105 


113 


Students Tested In Reading 


Oct-95 MAT/7 (a) 


92 


83 


Apr-96 Stanford 


96 




Oct-96 MAT/7 (a){b) 




83 


Apr-97 ITBS 


102 




Oct-97 MAT/7 


82 


80 



112 

113 

118 



83 




82% 


74% 


74% 


89 




86% 




79% 


94 


58 


89% 


73% 


84% 


81 




78% 


71% 


69% 



51% 



Source: Detailed results from the test publishers provided by the Edison Project. 

On the other hand, given the 20 to 25 percent exclusion rate from testing, the math scores 
in grades 3 and 4 of 1995, which constitute the math baseline, are improbably low. Also 
surprising is the fact that they are lower than scores at the impoverished school run by Edison in 
Dade County, Florida, where 90 percent of students qualified for free or reduced-price lunch, in 
comparison to only 5 1 percent in the Boston Edison. 

• The 1995 grade 3 math score of 37 NCEs compared with 40 NCEs in Dade County. 

• The 1995 grade 4 math score of 29 NCEs is less than the 33 NCE level in Dade County. 
We have no way of knowing why the Boston baseline scores were so poor, especially given the 
boost that excluding a relatively large number of students would give to the scores. However, 
there is no question that a lower baseline made the achievement of Edison students look stronger 
when they were tested the next year. 



3. Massachusetts grade 3 reading test. The final test administered to students at 
Edison's Boston charter school for which we currently have data is the ITBS (Iowa Test of Basic 
Skills) reading test administered to third graders in spring 1997. It is part of the statewide 
assessment program that replaced MEAP in 1997. As Table F5 shows, only 6 percent of Edison 
students were not tested in this state-mandated and monitored exam, compared to the 20 or 25 
percent when Edison admininstered its own test. Even then, Edison tests fewer students than the 
comparison districts. The comparison schools in Table F5 and Figure F2 are the same school 
districts used by Edison in their year-end performance reports. Chelsea was “taken over” by the 
state several years ago, and its educational and economic problems are well documented. 

As for the test results, third graders scored at the 47-NCE level, compared to the Boston 
public school average of 43. However, they had not advanced beyond the level that they had 
already attained on the MAT/7 when the school first opened in fall 1995 (see Table B). While an 
NCE of 45 appears close to the national norms, the state average score was 55 NCEs. 
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Table F5 

Massachusetts 1997 Third-Grade Reading Assessment 



Students Tested Under Routine Conditions (a) 

Percent 





Operating 

Educ. 

$/Pupil 


Percent 

Non- 

White 


Absent 
For Test 


National 

Norms 

%tile 


National 

Norms 

NCE 


Percent 

Pre- 

Readers 


Proficient 

or 

Advanced 


State Average 


5.468 


22% 




60 


55 


4% 


75% 


Edison-Boston 


6,464 


76% 


6% 


45 


47 


10% 


46% 


Boston Public 


6,953 


83% 


5% 


36 


43 


13% 


45% 


Chelsea 


5,068 


82% 


1% 


30 


39 


18% 


38% 


Somerville 


5,959 


40% 


0% 


46 


48 


10% 


56% 


Lowell 


4,799 


55% 


3% 


38 


44 


12% 


51% 


Brockton 


4,983 


51% 


1% 


39 


44 


13% 


50% 


Worcester 


5,409 


43% 


1% 


57 


54 


7% 


68% 


Edison- 


na 


48% 


0% 


40 


45 


18% 


40% 



Worcester 

(a) Many special education and LEP students are excluded from testing under 
"routine conditions." 



Figure F2 Massachusetts 1997 Grade 3 Reading Test (ITBS) 

g 80% 

I 70% 
g 60% 

I 50% 
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Edison talks up student achievement in its Boston charter, and given some of the 
characteristics of its student body, in comparison with Edison’s other schools and the Boston 
public schools serving similar students, you would expect it to make a good showing. Only 
about half of its students qualify for free or reduced-price lunch, and it has a lower percentage of 
special education students than schools in surrounding districts. Edison reports test results for 
many fewer students than urban school districts in Massachusetts or than other Edison schools. 
This, too, should give it an advantage. But despite these facts—and other miscellaneous attempts 
to make scores look better than they are— once you make your way through the confusion of test 
scores, it turns out that Boston’s Edison charter is about average. In fact, it is still achieving at 
about the level of other schools serving low-income students that are not participating in the 
Edison program. 
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VIII. First-Year Achievement in Edison Schools Opened in 1996 



Based on the experience of its first four schools, which opened in 1995 (in Boston, 
Wichita, Mt. Clemens, and Sherman), Edison has argued that its students start off as low 
achievers, like those in most schools serving disadvantaged children. But after one year of the 
Edison program, students attain scores exceeding the national average. In fact, as this report has 
shown, results fi"om Edison's first four schools are mixed and inconclusive. Some data support 
the Edison claim in Wichita and Boston, though not results fi"om state assessments. And early 
results fi-om the nine schools that Edison opened in 1996 are equally inconclusive.^^ 

Only three of the new schools have fall baseline data: Wichita Jimior Academy 
(MAT/7); and Colorado Springs (district tests); or pre-Edison baseline data (Dade Coimty). So 
we will not be able to see where students fi-om the other Edison schools were in terms of 
achievement when they entered the Edison program. However, spring assessments were given in 
the other schools, and the results offer a first look at student achievement under Edison and invite 
a comparison to the achievement in the four schools Edison opened in 1995. 

In Colorado Springs, fall-to-spring improvement in the first year was less than the 
district's and less than when the Edison school was a traditional neighborhood school (see the 
discussion in Section VII above). Spring testing results for the other new elementary schools are 
shown in Table VIII-l, along with results for three of the first Edison schools. Results for 
middle schools in Worcester, Mt. Clemens, and Wichita are shown in Table VIII-2. 

The two extremes are Lansing, where students are achieving at national norms, and Dade 
County, which has the lowest scores registered by any of the Edison schools (Table VIII- 1). Mt. 
Clemens’ sixth and seventh graders scored just above the 40* percentile in reading and just 
below the 40* percentile in math and language. Eighth graders scored lower. Students at the 
Worcester charter school scored around the 40* percentile in reading and language at most grade 
levels and in the 30* percentile range in math (Table VIII-2). 



They are elementary schools in Colorado Springs (see section VI), Dade County, Fla. (see section VI), Worcester, 
Mass., and Lansing, Mich.; and middle schools in Boston, Mt. Clemens and Lansing, Mich., Wichita, and Sherman, 
Texas. 
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Table VIII-1 

First-Year Achievement in Year-Two Elementary Schools 
Compared to Year-One Schools 
(Normal Curve Equivalents) 



Year-One Schools 





Edison-Worcester 
Test Date NCE 


Dade 
County 
Stanford 
Spring 97 


Mid- 

Michigan 
MAT/7 
Spring 97 


Boston- 

Charter 

Stanford 

Apr-96 


Wichita 

MAT/7 

Oct-96 


Mt 

Clemens 

ITBS 

Dec-96 


Reading 


















Grade 3 


ITBS 


Apr-97 


45 












Grade 3 


Stanford 


Jun-97 


45 


31 


49 


52 


48 


37 


Grade 4 


Stanford 


Jun-97 


43 


31 


49 




49 


39 


Grade 5 


MAT/7 


Apr-97 


43 


35 




56 


55 


46 


Math 


















Grade 3 


Stanford 


Jun-97 


34 


33 


50 


54 


54 


35 


Grade 4 


Stanford 


Jun-97 


37 


40 


48 




51 


41 


Grade 5 


MAT/7 


Apr-97 


43 


38 




50 


58 


33 


Language 


















Grade 3 


Stanford 


Jun-97 


47 




50 


53 




38 


Grade 4 


Stanford 


Jun-97 


39 




49 






41 


Grade 5 


MAT/7 


Apr-97 


43 






54 




45 



Source: Data provided to the AFT by Edison Project. 



Table VIII-2 

First-Year Achievement In Worcester, Boston, Dade County and 
Mt. Clemens Middle Schools 

Normal Curve Equivalents 

Mt. 

Boston Clemens Wichita Wichita 
Edison-Worcester Stanford ITBS MAT/7 MAT/7 
Test Date NCE Jun-97 Apr-97 Oct-96 Oct-97 

Reading 



Grade 6 


MAT/7 


Jun-97 


39 


53 


46 


51 


52 


Grade 7 


MAT/7 


Jun-97 


46 


45 


48 


45 


50 


Grade 8 








41 


35 


46 


47 


Math 
















Grade 6 


MAT/7 


Jun-97 


32 


42 


40 


43 


43 


Grade 7 


MAT/7 


Jun-97 


43 


34 


43 


45 


46 


Grade 8 








38 


34 


44 


43 


Language 
















Grade 6 


MAT/7 


Jun-97 


41 


no test 


42 


no test 


no test 


Grade 7 


MAT/7 


Jun-97 


45 


no test 


43 


no test 


no test 


Grade8 






no test 


no test 


35 


no test 


no test 



Source: Data provided to the AFT by Edison Project. 
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There is no apparent pattern to these variations in achievement, even in districts where 
Edison opened a second school in 1996. The Mt. Clemens middle school is doing better than the 
elementary school, which opened in 1995. The new Boston and Wichita schools are doing less 
well (cf. Tables VIII- 1 and VIII-2). In the case of the Sherman middle school, questions about 
the student population make it difficult to be certain (see discussion below). 

Edison's Wichita elementary school, which opened in 1995, is unquestionably its most 
successful school. In the fall of its second year (measuring about one year’s progress), fifth 
graders scored above the national average, while third and fourth graders scored around the 
national average. However, students in Edison's Wichita middle school, for which we have two 
years of data, show no change in student achievement, which is more like that of the control 
schools. 

The Edison middle school has spring Kansas state assessments in grade 7 reading and 
math (see Table VIII-3 and Figure VIII-1). It is essentially a neighborhood school, and the 
percentage of students qualifying for free or reduced-price lunch is only a fraction less than the 
pre-Edison school. Along with Edison, the control schools comprise the five poorest middle 
schools in Wichita. 

The Edison school is below the state average in student achievement— as are the control 
schools (see Table Vni-3)— and somewhat below the district average. The spring 1997 scores for 
Hamilton, the poorest school, are similar to Edison's, except that Hamilton registered 
improvements in expository reading and math (see Figure VIII-1 and Table VIII-3). Whatever 
effect Edison’s longer school year has will not be picked up in the 1997 comparison. 
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Figure VIIM Kansas State Grade 7 Expository Reading Test 




Hamilton Marshall Jardine- Pleasant Wichita Curtis 

Edison Valley Average 



Tabie Vlil-3 . 

Grade 7 Kansas State Assessments at Jardine-Edison in Wichita 

Edison Pleasant 



State 


Wichita 


Jardine 


Jardine 


Curtis 


Hamilton 


Marshall 


Valley 


% White (95-96) 




55% 




49% 


55% 


41% 


51% 


% Free Lunch (95-96) 




77% 




53% 


77% 


63% 


59% 


% Free Lunch (97-98) 






74% 


63% 


79% 


68% 


67% 


Teachers/100 Pupils (95-96) 
Grade 7 Expository Reading 


6.43 


6.44 


4.70 


6.09 


6.43 


7.03 


5.95 


April, 1995 59 


48 


43 




48 


43 


46 


47 


April. 1996 59 


49 


46 




53 


38 


47 


51 


March, 1997 


52 




44 


56 


42 


43 


51 


One- /ear Change 

Grade 7 Narrative Reading 


3 




-2 


3 


4 


-4 


0 


March, 1995 61 


54 


51 




57 


49 


52 


56 


March. 1996 


56 




50 


56 


47 


51 


53 


One-Year Change 
Grade 7 Math Power 


2 




-1 


-1 


-2 


-1 


-3 


April, 1995 51 


43 


37 




43 


38 


38 


44 


April. 1996 52 


44 


34 




46 


35 


43 


45 


April. 1997 


43 




36 


43 


39 


39 


41 


One-Year Change 


-1 




2 


-3 


4 


-4 


-4 



Source: School reports on WPS Web page (http://www.feist.com/-wpsweb/), and information provided by 
school district officials. 

In contrast to the success of Edison's elementary school on the MAT/7, the Edison middle 
school in Wichita showed no improvement in math and only a little in reading during its first 
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year. The average reading improvement of 2.3 NCEs exceeded the district average increase and 
the increase in the four control schools (see Figure VIII-2 and Table VIII-4). In math, however, 
the district and all of the control schools had gains that exceeded Edison's. Overall, the changes 
at the Edison middle school, at the control schools, and in the district average were very small 
and essentially not different from each other. 



Figure VIII-2 One Year of Progress in Math and 
Reading (MAT/7) at Jardine-Edison in Wichita 




Grade 6 Grade 7 Grade 8 Grade 6 Grade 7 Grade 8 



.^niirrp Tp.Qt niihli.Qhpr*.Q data nrnvidpd hv thp Fdi.Qnn Prniprt 



Table VIII-4 

MAT/7 Standardized Achievement Test at Jardine-Edison in Wichita 

Edison Pleasant 





Wichita 


Jardine 


Jardine 


Curtis 


Hamilton 


Marshall 


Valley 


% White (95-96) 




55% 




49% 


55% 


41% 


51% 


% Free Lunch (95-96) 




77% 




53% 


77% 


63% 


59% 


% Free Lunch (97-98) 






74% 


63% 


79% 


68% 


67% 


Teachers/100 Pupils (95-96) 


6.43 


6.44 


na 


6.09 


6.43 


7.03 


5.95 


MAT? Reading (NCEs) 
Grades 6-8, Fall 1 995 


54.3 


48.0 




53.3 


48.3 


49.3 


55.0 


Grades 6-8, Fall 1996 


53.3 




47.3 


52.7 


44.3 


48.7 


52.7 


Grades 6-8, Fall 1997 


53.7 




49.7 


52.0 


44.7 


48.3 


53.3 


One-Year Change 


0.3 




2.3 


-0.7 


0.3 


-0.3 


0.7 


MAT7 Math (NCEs) 
Grades 6-8, Fall 1995 


48.7 


43.7 




47.3 


41.3 


42.0 


50.0 


Grades 6-8, Fall 1 996 


50.7 




44.3 


49.3 


40.3 


44.0 


48.3 


Grades 6-8, Fall 1 997 


52.3 




44.3 


50.7 


41.3 


46.7 


51.0 


One-Year Change 


1J 




0.0 


13 


10 


2.7 


2.7 



Source: School reports on WPS Web page (http://www.usd259.com/rniddle/rniddle.htrnl), and information 



provided by school district officials. 
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Evaluating the achievement of Edison’s middle school in Sherman, Texas, is a much 
bigger challenge. In 1995, Edison began operating the 460-student Washington elementary 
school in Sherman, Texas. One year later, Edison opened a school-within-a-school in the same 
district for grades 5 and 6. Edison's operation enrolls 300 of the 900 students in the school; and 
state records, including the accoimtability system, combine the records of Edison and non-Edison 
students. 

The Sherman elementary school has not been a success so far, and Edison admits as 
much. However, the intermediate school was coimted twice in the list of successes in Edison's 
Annual Report of School Performance for improvement in both fifth and sixth grade 
(http://www.edisonproiect.com/annualfratne.html) . According to the report, “the comparisons show 
Edison with more significant gains than Sherman's traditional program in both reading and math 
in both fifth grade and sixth. This is a strong start indeed.” 

Edison's language implies both: 1) significant gains by Edison students, and 2) 
significant differences between Edison and traditional students. In fact, given the data we have, 
both of these claims are questionable at best. 

The scores at Edison's middle school are based on the Texas Learning Index (TLI). TLI is 
not just a restatement of the TAAS. It is a standardized score based on the actual TAAS scores 
rather than the percent passing, which is how TAAS scores are usually rendered. It ranges fi"om 
0 to 100 and allows the comparison of different students in the same year or the comparison of 
different years for the same students. In other words, it allows student and school scores to be 
looked at in a number of different ways. An important peculiarity of the TLI is that it does not 
include students who score above 84 because of a so-called ceiling effect. 

The definition of significance that Edison employs in dubbing its Sherman middle school a 
success does not involve comparing average scores or gains over the course of the year, as would 
be quite possible, given the way TLI is set up. Instead, it depends on a “difference in percentage 
proficient scores of 5 percentage points or more.” Edison's figures go as follows: 

• 33 percent of Edison fifth graders improved their TLI in reading by 5 points, compared with 
25 percent of the traditional students; and 36 percent of Edison fifth graders improved by 5 
points in math compared to 25 percent of the traditional students. 

• 45 percent of Edison sixth graders improved their TLI in reading by 5 points compared with 
39 percent of other students; and 47 percent improved by 5 points in math compared to 36 
percent of traditional students. 

The figures meet Edison's own criterion. However, given the nature of the TLI, as well as the 
Texas state figures about Dillingham, there are some unanswered questions. 

According to the state of Texas, only 63 percent of the Dillingham fifth and sixth graders 
(Edison and traditional students combined) increased their TLI reading scores by 5 or more 
points and 49 percent increased their TLI math score by at least 5 points 
(http://www.tea.state.tx.iis/perfrepoit''ciy'97/group.srch.html) . These figures don't track with the Edison 
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figures, and they seem much lower. Furthermore, in the 900-student school (both Edison and 
traditional students) the TLI included only 316 students in reading and 437 students in math. 

Did Edison include some Edison students who scored above the cutoff of 84 in computing its 
five-point gains? We can't tell. 

The ceiling effect is likely to have a much bigger impact on traditional students because they 
have, on average, higher scores than the Edison students. Based on the spring 1997 TAAS 
scores that the company provided to the AFT and the data for the entire 900-student school 
obtained fi"om the Internet (http://vvw\v.tea.state.tx.us/perfreport/aeis/97/canipiis.srch.html) . Edison 
students are performing well below traditional students in reading and math: 

• Edison fifth-grade students averaged a TAAS passing score of 78 percent, compared to 
91 percent for traditional students. 

• In math, they averaged a 71 percent pass rate, compared to an 89 percent for traditional 
students. 

• In sixth-grade reading, the pass rate for Edison students was 82 percent, compared to 93 
percent for traditional students. 

• Based on improvement, the entire student body of 900 ranked in the top quartile in the 
state in 1996-97. 

Did Edison get the favorable improvement rates for Edison students because the ceiling effect 
eliminated a number of higher-scoring traditional students? Again, we don't know because 
Edison's figures are unconnected to any of the other test data available through the TLI. 

In order for Edison to claim its Sherman middle school as a success, it needs to provide 
demographic and student achievement data on its own 300 students and the 600 students in the 
traditional part of the school. The TLI data should be presented using the same rules that the 
state accountability system uses to account for the ceiling effect. Ideally, 300 similar students 
fi'om the traditional school could be selected as a control group. Student achievement gains (or 
declines) need to be calculated for both years. The analysis presented in this section, while more 
extensive and accurate than Edison's, is certainly incomplete, but it suggests that Edison students 
are lagging behind in math and reading. 

There is no reason why we should be able to generalize about student achievement in the 
schools Edison opened in 1996. Looking at the record, the most we can say with certainty is that 
some schools do better than others— which suggests that schools are anything but turnkey 
operations. And Edison's program, like any other, depends on the people who have to carry it 
out. If school districts expect magic, they are going to be disappointed. 
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Recommendations 



As one might expect at this early stage in the history of the Edison Project, the evidence 
on student achievement is mixed and inconclusive. However, two things are clear. There are 
discrepancies between the record of Edison schools, as measured by standard methods of 
educational evaluation, and the company’s sales presentations and promotional materials. 

Edison has exaggerated test score gains and emphasized favorable comparisons to show Edison 
schools in the most positive light. In fact, if public schools were to use some of Edison’s 
evaluation methods and modes of presenting data, they would look a lot better, too. 

This is unacceptable. Edison should be expected to measure achievement in its schools 
using the same standards that apply to other schools in the districts where the company operates. 
Despite its claims to being a better alternative to regular public schools and a model for public 
education, Edison is obviously confronting the same difficulties in improving student 
achievement as regular public schools. 

Although accountability for the outcomes of private management of a public institution, 
like a school, rests with the private provider, the ultimate responsibility is with public officials 
whose duty it is to protect the public interest. Private management of public schools requires 
good public oversight, both before entering into an agreement with a private provider and during 
it. To that end, there are a number of things school districts that are considering hiring the 
Edison Project should keep in mind: 

• Improving student achievement is at the heart of all the many efforts to improve U.S. 
education, including Edison’s. So school districts should pay at least as much attention to 
Edison’s student achievement data as they do to impressions from site visits to Edison 
schools. 

• School districts should have their own experts— district officials in charge of program 
evaluation and student assessment— review and verify ^ student achievement data in ^ of 
the Edison schools. This review should be made public and should include not only Edison- 
administered achievement tests but also state and local assessments. 

• School districts considering hiring Edison should have a timeline that is unhurried enough so 
that everyone involved can give careful attention to the advantages and disadvantages of 
Edison before making a decision. 

• If a school district decides to sign a contract with Edison, officials should either hire 
independent professional evaluators or conduct their own annual evaluation of program 
implementation and student achievement using qualified school-district employees. 

In her article about Edison’s Boston charter school, which appeared in the March 1998 
issue of Phi Delta Kappan, Peggy Farber puts the Edison Project into its perspective as a 
business venture and as one struggling school reform project among many. She directs her 
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caveats to members of the media, but they are equally applicable to school districts: 

It is not reasonable to expect the Edison Project to present the public--its potential 
customers— with a truly comprehensive, objective picture of its schools. Edison officials 
are naturally eager to draw attention to signs of success, which clearly exist. But it is 
essential, especially now, when the idea of a single solution to complex social problems 
has such a strong grip on the American imagination, that reporters give an honest and 
thorough accoimting of what it's like inside schools that operate beyond the reach of 
almost all local and state agencies. The evidence from Boston Renaissance suggests that 
the Edison Project is struggling— succeeding in some areas, stumbling in others—to 
improve schools for students. And the same can be said for coimtless reform efforts 
across the coimtry. 

To a large extent, Edison’s popularity depends on the perception that public schools are 
dysfunctional and helpless to change and that anyone with a fix to offer must be able to do better. 
Before buying into this idea, public officials and the people who report and comment on 
education should look at some of the reform efforts that are demonstrably-that is, using accepted 
measures 

—changing public schools for the better. This report has had occasion to comment on one of 
them-Success for All— and to point out that, thus far, Edison students who are following the SEA 
program are not doing as well as SFA students in non-Edison schools where SFA is fully 
implemented. This is true despite the fact that Edison students are generally more advantaged 
and have the benefit of the rest of the Edison program. 

The point is that Edison is one reform effort among many. Its more polished public 
relations have ensured it a national reputation, and since it is a for-profit business, it also has 
more money behind it. Will the Edison Project be able to live up to its promises? We’ll have to 
wait and see. In the meantime, there are unheralded programs that are already living up to their 
claims, and people who are looking for ideas— or good news about education— should be giving 
them the same kind of attention that Edison gets. 
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Appendix: Sherman's Status as a Low-Performing School 

The state of Texas placed the Edison school on its list of low-performing schools for part 
of 1997 due to the low performance of its Hispanic students on the math part of TAAS. Of 
nearly 6,000 schools in Texas, only about 70 are classified as low performing. (See 
http://www.tea.state.tx.us/accountabilitv/rat97/ for details of the Texas Accountability Rating System.) 
The school was removed from this list when seven Hispanic third graders were labeled as 
limited-English-proficient well after the test was taken. The action resulted in an exemption of 
all of Sherman’s Hispanic students from the state’s accountability plan. 

Among the criteria by which schools can be classified as low performing are pass rates 
below 35 percent on any TAAS test in any one of four categories of students; 1) white, 2) 
African-American, 3) Hispanic, or 4) economically disadvantaged. However, 30 students must 
be in each category for the score to count. Table C3 shows pass rates for three categories of 
students in June 1997. 

Table C3 

TAAS Passing Rate by Category of Student 
Edison School in Sherman 



Economically 

Hispanic African-American Disadvantaged 





94-95 


95-96 


96-97 


94-95 


95-96 


96-97 


94-95 


95-96 


96-97 


Gr^ade 3 




















Reading 


78% 


41% 


50% 


100% 


84% 


50% 


83% 


61% 


65% 


Math 


69% 


56% 


14% 


71% 


46% 


39% 


62% 


65% 


55% 


# Tested 






14 






10 






49 


Grade 4 




















Reading 


50% 


33% 


59% 


11% 


55% 


33% 


50% 


53% 


66% 


Math 


50% 


33% 


41% 


44% 


60% 


25% 


57% 


27% 


48% 


Writing 


70% 


38% 


59% 


50% 


60% 


55% 


83% 


60% 


46% 


# Tested 






17 






9 






44 


# Tested (Gr. 3 & 4) 




31 






19 
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Source: Campus Reports of Academic Excellence Indicator System , Division of 
Performance Reporting, Texas Education Agency. 



The 14 percent pass rate in third grade math in 1996-97 is attention getting. It contrasts 
sharply with the 50 percent pass rate in reading (one would normally expect LEP students to do 
better in math than in reading) and the math passing rate from the prior year of 56 percent. The 
31 Hispanic students (14 in third grade and 17 in fourth grade) had an average passing rate of 29 
percent (14 percent in third grade and 41 percent in fourth grade). As a result of falling below 
the 35 percent standard in the area of “Hispanic math,” the Edison school was put on the low- 
performing schools list. When the seven Hispanic students were subsequently exempted from 
the accountability criteria, dropping total Hispanic students with countable test scores to 24, 
Edison was removed from the list. As noted above, low performance of Hispanic students only 
counts under the Texas Accountability Rating system if at least 30 Hispanic students are tested. 
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